- 04 9月, 2020 1 次提交
-
-
由 Boyang Jerry Peng 提交于
* Various fixes for Batch Source 1. Create intermediate topic/subscription prior to function running in case auto topic creation is turned off 2. Fix possible NPE in CronTrigger when calling stop() 3. Stop all producers created in ContextImpl 4. Delete intermediate topic for batch source Co-authored-by: NJerry Peng <jerryp@splunk.com>
-
- 30 8月, 2020 1 次提交
-
-
由 Boyang Jerry Peng 提交于
Co-authored-by: NJerry Peng <jerryp@splunk.com>
-
- 28 8月, 2020 1 次提交
-
-
由 Boyang Jerry Peng 提交于
Co-authored-by: NJerry Peng <jerryp@splunk.com>
-
- 21 8月, 2020 1 次提交
-
-
由 Boyang Jerry Peng 提交于
Motivation close() method never gets called in BatchSource
-
- 10 8月, 2020 1 次提交
-
-
由 Sanjeev Kulkarni 提交于
If a function has many threads who are doing context.newOutputMessage().send(), there could be a lot of contention due to big synchronization blocks in the ProducerImpl. By allowing functions to use thread local producers, this synchronization can be avoided leading to increased performance. This pr adds the configurability of using thread local producers in functions and sources Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>
-
- 05 8月, 2020 1 次提交
-
-
由 Sanjeev Kulkarni 提交于
* Added ability to specify producer config for functions and sources * Fixed test * Fix test * Add generated function proto * Add header * Address comments Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>
-
- 04 8月, 2020 1 次提交
-
-
由 Boyang Jerry Peng 提交于
Co-authored-by: NJerry Peng <jerryp@splunk.com>
-
- 25 7月, 2020 3 次提交
-
-
由 Sergii Zhevzhyk 提交于
* [Issue 7489] Remove timestamp from exception metrics for functions and connectors * [Issue 7489] Remove timestamp from exception metrics for Go functions * [Issue 7489] Remove unused import in go stats * [Issue 7489] Remove timestamp from metrics in python stats * Change to v2 of go github actions * Update go github actions * Remove the version from the go test command * Rename github jobs for go Co-authored-by: NMatteo Merli <mmerli@splunk.com>
-
由 Sanjeev Kulkarni 提交于
* Allow ability to specify retain key ordering in functions * Address comments Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>
-
由 Sanjeev Kulkarni 提交于
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>
-
- 24 7月, 2020 1 次提交
-
-
由 Sergii Zhevzhyk 提交于
-
- 17 7月, 2020 1 次提交
-
-
由 冉小龙 提交于
## Motivation Provide support to configure different docker images for go, python, java function ## Modifications Add javaFunctionDockerImageName, pythonFunctionDockerImageName, and goFunctionDockerImageName in KubernetesRuntimeFactoryConfig. Add unit test case Update docs of functions-runtime.md
-
- 02 7月, 2020 1 次提交
-
-
由 feynmanlin 提交于
Fixes #5538 ### Motivation ### Modifications In function mode and sink mode, PulsarSource can read compacted topic。 By `inputSpecs` parameter, each topic can independently decide whether to read compacted ### Verifying this change unit test: org.apache.pulsar.io.PulsarFunctionE2ETest#testReadCompactedFunction org.apache.pulsar.io.PulsarFunctionE2ETest#testReadCompactedSink
-
- 09 6月, 2020 1 次提交
-
-
由 feynmanlin 提交于
1) Change the value of the CLI tool parameter "--custom-schema-input" from "TopicName-> schemaType" to "topicName-> {" schemaType ":" type"," jsr310ConversionEnabled ": true," alwaysAllowNull ": true}" 2) Modify Function.proto, add properties "jsr310ConversionEnabled", "alwaysAllowNull"。So that we can receive the above 2 parameters 3) Modify the "FunctionConfigUtils#convert" method , put the two parameters in "CustomSchemaInputs" into ConsumerSpec 4) In “JavaInstanceRunnable#setupInput” method, put the above 2 parameters into "ConsumerConfig" and pass it to "PulsarSource", let it set the parameters into "schema" when creating the consumer of Source。 So that,The “function” can get these 2 parameters from the message of ”currentRecord“
-
- 30 5月, 2020 2 次提交
-
-
由 feynmanlin 提交于
Fixes #7032 ### Motivation Source data flow: Custom Source-> Function-> Pulsar producer-> Broker Sink data flow: Broker-> Pulsar consumer-> Function-> Custom Sink Pulsar consumer is only used in sink mode, so it is necessary to add DLQ parameters for sink mode
-
由 Sanjeev Kulkarni 提交于
* Moved ClassLoading and Reflections Helper functions to common * Fix tests * Fix test Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>
-
- 23 5月, 2020 1 次提交
-
-
由 Sanjeev Kulkarni 提交于
* Fix null pointer when getting function instance metrics. * Made more functions sync * Made the remaining public interface synchronized * Made stats class public method synchronized so they are thread safe. * Made setup synchronized so that it and close won't run together * Undo making stats sync until we resolve differences * Incorporated feedback Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>
-
- 19 5月, 2020 1 次提交
-
-
由 Sanjeev Kulkarni 提交于
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>
-
- 12 5月, 2020 1 次提交
-
-
由 Sanjeev Kulkarni 提交于
* Make Nar Extraction Directory configurable * Fixed unittests Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>
-
- 08 5月, 2020 1 次提交
-
-
由 Jia Zhai 提交于
Fixes #6519 ### Motivation Currently, Pulsar Functions not support Async mode, e.g. user passed in a Function in format : ``` Function<I, CompletableFuture<O>> ``` This kind of function is useful if the function might use RPCs to call external systems. e.g. ```java public class AsyncFunction implements Function<String, CompletableFuture<O>> { CompletableFuture<O> apply (String input) { CompletableFuture future = new CompletableFuture(); ...function compute... future.whenComplete(() -> { ... call external system ... }); return future; } ``` ### Modifications - add support for Async Functions support. ### Verifying this change current ut passed. * support func: Function<I, CompletableFuture<O>> * add 2 examples * add limit to the max outstanding items
-
- 30 4月, 2020 1 次提交
-
-
由 guangning 提交于
Fixes #5633 ### Motivation Currently, some users want to support Avro schema in debezium, so this pr supports this feature. For Kafka's Avro schema, it depends on the Avro 1.8 version, but Avro version has just been upgraded to 1.9 in pulsar, so shade is needed to avoid finding `addProp` function ### Modifications * Add a package `kafka-connect-avro-converter-shaded` * Add class KafkaSchema to converter Kafka's Avro schema to pulsar's schema ### Verifying this change Unit test and integration tests
-
- 22 4月, 2020 1 次提交
-
-
由 Sanjeev Kulkarni 提交于
Co-authored-by: NSanjeev Kulkarni <sanjeevk@splunk.com>
-
- 05 4月, 2020 1 次提交
-
-
由 ran 提交于
Fixes #5503 From #6445 # Motivation In functions, it will encounter ```ClassCastException``` when using the Avro schema for topics. ``` Exception in thread "main" java.lang.ClassCastException: org.apache.pulsar.shade.org.apache.avro.generic.GenericData$Record cannot be cast to io.streamnative.KeyValueSchemaTest$Foo2 at io.streamnative.KeyValueSchemaTest.testConsumerByPythonProduce(KeyValueSchemaTest.java:412) at io.streamnative.KeyValueSchemaTest.main(KeyValueSchemaTest.java:305) ``` # Modifications In functions, when using Avro schema specific the ClassLoader for ReflectDatumReader. Add integration test ```testAvroSchemaFunction``` in class ```PulsarFunctionsTest```.
-
- 31 3月, 2020 1 次提交
-
-
由 guangning 提交于
### Motivation At present, in the test, we found that due to the addition of a field `forwardSourceMessageProperty` in the proto file of function, this field was lost in the proto file generated by python and go. Due to python parsing with the following code: ``` json_format.Parse(args.function_details, function_details) ``` the following exception will be thrown. ``` 2020-03-30T13:13:25.2339031Z 13:13:24.379 [pulsar-external-listener-20-1] INFO org.apache.pulsar.functions.runtime.process.ProcessRuntime - Started process successfully 2020-03-30T13:13:25.2339190Z Traceback (most recent call last): 2020-03-30T13:13:25.2340782Z File "/pulsar/instances/python-instance/python_instance_main.py", line 211, in <module> 2020-03-30T13:13:25.2340944Z main() 2020-03-30T13:13:25.2342589Z File "/pulsar/instances/python-instance/python_instance_main.py", line 98, in main 2020-03-30T13:13:25.2342744Z json_format.Parse(args.function_details, function_details) 2020-03-30T13:13:25.2354119Z File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 430, in Parse 2020-03-30T13:13:25.2354284Z return ParseDict(js, message, ignore_unknown_fields, descriptor_pool) 2020-03-30T13:13:25.2354689Z File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 450, in ParseDict 2020-03-30T13:13:25.2354882Z parser.ConvertMessage(js_dict, message) 2020-03-30T13:13:25.2355386Z File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 481, in ConvertMessage 2020-03-30T13:13:25.2355537Z self._ConvertFieldValuePair(value, message) 2020-03-30T13:13:25.2356082Z File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 590, in _ConvertFieldValuePair 2020-03-30T13:13:25.2356559Z raise ParseError('Failed to parse {0} field: {1}.'.format(name, e)) 2020-03-30T13:13:25.2357199Z google.protobuf.json_format.ParseError: Failed to parse sink field: Message type "proto.SinkSpec" has no field named "forwardSourceMessageProperty". 2020-03-30T13:13:25.2357634Z Available Fields(except extensions): ['className', 'configs', 'typeClassName', 'topic', 'serDeClassName', 'builtin', 'schemaType']. ``` This pr is mainly to fix the proto file generated by python first. In order to make the test pass smoothly, I will gradually fix this problem in other languages in the next pull request. ### Modifications * Enable build docker image to pulsar and pulsar-all. * Add new generated protobuf file for python * Disable go function integration test * Add sleep for method testGetPartitionedStatsInternal ### Verifying this change The integration process test passed https://github.com/AmateurEvents/pulsar/pull/22
-
- 28 3月, 2020 1 次提交
-
-
由 Yijie Shen 提交于
This PR aims to fix or simplify `Optional` type usage logic: - Null value used for optional type. - Optional.isPresent() can be replaced with functional style expression - Optional call chain that can be simplified.
-
- 27 3月, 2020 2 次提交
-
-
由 Yijie Shen 提交于
-
由 Yijie Shen 提交于
-
- 26 3月, 2020 2 次提交
-
-
由 Yijie Shen 提交于
-
由 Neng Lu 提交于
Fixes #6170 ### Motivation Allow users set custom system properties while submitting functions. This can be used to pass credentials via system property. ### Modifications - pulsar-admin client tool to accept multiple `--custom-property` options while creating the function - function's java runtime to append these provided properties.
-
- 06 3月, 2020 1 次提交
-
-
由 congbo 提交于
### Motivation Master Issue: #5454 When one Consumer subscribe multi topic, setSchemaInfoPorvider() will be covered by the consumer generated by the last topic. ### Modification clone schema for each consumer generated by topic. ### Verifying this change Add the schemaTest for it.
-
- 20 2月, 2020 1 次提交
-
-
由 Neng Lu 提交于
-
- 16 2月, 2020 1 次提交
-
-
由 Devin Bost 提交于
[Issue 4175] [pulsar-function-go] Create integration tests for Go Functions for production-readiness (#6104) This PR is to provide integration tests that test execution of Go functions that are managed by the Java FunctionManager. This will allow us to test things like behavior during function timeouts, heartbeat failures, and other situations that can only be effectively tested in an integration test. Master issue: #4175 Fixes issue: #6204 ### Modifications We must add Go to the integration testing logic. We must also build the Go dependencies into the test Dockerfile to ensure the Go binaries are available at runtime for the integration tests.
-
- 14 2月, 2020 1 次提交
-
-
由 Neng Lu 提交于
Fixes #5116 ### Motivation Based on the request in #5116, adding this flag can help reduce ambiguity and increase flexibility. By default, the function passes source message properties. ### Modifications - add flags in pulsar admin client tools - update functions proto and functionConfig to contain the flag - update the pulsar sink to pass source message property if the flag is set
-
- 25 1月, 2020 1 次提交
-
-
由 Sergii Zhevzhyk 提交于
-
- 21 1月, 2020 1 次提交
-
-
由 Sergii Zhevzhyk 提交于
### Motivation Some of the classes in the pulsar-functions module had a mixture of the following lombok annotations: ``` @data @Setter @Getter @EqualsAndHashCode @ToString ``` The [@data](https://projectlombok.org/features/Data) annotation includes all other annotations: > All together now: A shortcut for @ToString, @EqualsAndHashCode, @Getter on all fields, @Setter on all non-final fields, and @RequiredArgsConstructor! ### Modifications Removed `@Setter`, `@Getter`, `@EqualsAndHashCode`, and '@ToString' if the `@data` annotation was also present
-
- 20 12月, 2019 1 次提交
-
-
由 Fred Eisele 提交于
Fixes #5726 ### Motivation When running pulsar-io connectors and functions from the Intellij IDE some actions fail due to uncaught class-not-found throwables. The expectation being that the class is being dynamically loaded and only the ClassNotFoundException will occur if the class is not found. When the function is created or run with https://pulsar.apache.org/docs/en/functions-deploying/#local-run-mode this is indeed the case. When running under the control https://pulsar.apache.org/docs/en/functions-debug/#debug-with-localrun-mode as a gradle plugin the class may already be known and throw a NoClassDefFoundError. It seems to me that any time ClassNotFoundException is handled then NoClassDefFoundError should also be caught. ### Modifications Wherever there was a `catch (ClassNotFoundException ` I replaced it with `catch (ClassNotFoundException | NoClassDefFoundError ` . There were multiple cases where the ClassNotFoundException were handled e.g. the jar loader failed so the nar loader was used to handle the jar loader's failure.
-
- 08 12月, 2019 1 次提交
-
-
由 Rajan Dhabalia 提交于
### Motivation With [PIP-3](https://github.com/apache/pulsar/wiki/PIP-3:-Message-dispatch-throttling) , Pulsar broker already supports to configure dispatch rate-limiting for a given topic. Dispatch-throttling feature allows user to configure absolute dispatch rate based on current publish-rate for a given topic or subscriber, and broker will make sure to dispatch only configured number of messages to the consumers regardless current publish-rate or backlog on that topic. Current dispatch-rate limiting doesn't consider change in publish-rate so, increasing publish-rate on the topic might be larger than configured dispatch-rate which will cause backlog on the topic and consumers will never be able to catch up the backlog unless user again reconfigured the dispatch-rate based on current publish-rate. Reconfiguring dispatch-rate based on publish-rate requires human interaction and monitoring. Therefore, we need a mechanism to configure dispatch rate relative to the current publish-rate on the topic. ### Modification `set-dispatch-rate` cli have a flag `--relative-to-publish-rate` to enable relative dispatch throttling. ``` pulsar-admin namespaces <property/cluster/namespace> set-dispatch-rate --msg-dispatch-rate 1000 --relative-to-publish-rate ``` ### Note: I will add broker-level configuration and documentation into separate PR.
-
- 06 11月, 2019 1 次提交
-
-
由 冉小龙 提交于
Signed-off-by: Nxiaolong.ran <ranxiaolong716@gmail.com>
-
- 04 11月, 2019 2 次提交
-
-
由 即将奔跑的蜗牛 提交于
### Motivation - Adjust code style ### Modifications - Add the missing placeholder in log statement - Delete redundant `return clause` for void method
-
由 冉小龙 提交于
* Add subscribe position param for consumer of sink Signed-off-by: Nxiaolong.ran <ranxiaolong716@gmail.com>
-