1. 06 1月, 2020 1 次提交
  2. 10 12月, 2019 1 次提交
  3. 25 6月, 2019 1 次提交
  4. 15 5月, 2019 1 次提交
  5. 11 4月, 2019 1 次提交
  6. 02 4月, 2019 1 次提交
  7. 12 3月, 2019 1 次提交
    • F
      [pulsar-io] Add a Pulsar IO connector for Redis sink (#3700) · c5e3baaf
      Fangbin Sun 提交于
      ### Motivation
      
      This PR provides a built-in Redis sink Connector, in order to cache messages in Redis [key-value] pairs. This will effectively make Redis a caching system, which other applications can access to get the latest value.
      
      ### Modifications
      
      Add a new sub-module in the `pulsar-io` module.
      
      ### Verifying this change
      
      This change can be verified as follows:
      
      * deploy the redis sink connector with configuration file containing the following fields:
      
      ```
      configs:
          redisHosts: "localhost:6379"
          redisPassword: "redis@123"
          redisDatabase: "1"
          clientMode: "Standalone"
          operationTimeout: "3000"
          batchSize: "100"
      ```
      * start a redis instance with auth
      * send messages with `NotNull` key/value in the topic declared when deploying the connector
      * check in Redis if the message's key-value pairs have been stored in above database
      
      ### Documentation
      ```
      # Submit a Redis Sink
      $ bin/pulsar-admin sink create --tenant public --namespace default --name redis-test-sink --sink-type redis --sink-config-file examples/redis-sink.yaml --inputs test_redis
          
      # List Sink
      $ bin/pulsar-admin sink list --tenant public --namespace default
          
      # Get Sink Info
      $ bin/pulsar-admin sink get --tenant public --namespace default --name redis-test-sink
          
      # Get Sink Status
      $ bin/pulsar-admin sink status --tenant public --namespace default --name redis-test-sink
          
      # Delete the Redis Sink
      $ bin/pulsar-admin sink delete --tenant public --namespace default --name redis-test-sink
      ```
      c5e3baaf
  8. 04 3月, 2019 1 次提交
    • T
      [Issue 3275][pulsar-io]Support source and sink of flume (#3597) · 37a2fda6
      tuteng 提交于
      * Support TLS authentication and authorization in standalone mode
      
      * Compile success
      To do: test channel, sink and source of flume
      
      * Add conf file
      
      * Add sink and source folder
      Move file to folder
      Add source
      Compile success
      To do -> test source
      
      * test flume source paas
      
      * Add config file and test case
      
      * Add test and update pom.xml
      To do add test of source and sink
      
      * Add unit tests
      
      * Add test case and test pass
      To do test source
      
      * Add license
      Add test source of pulsar
      
      * Handle if blockingQueue is null
      
      * Move LOG to log
      
      * Format code
      
      * Add sinkClass in pulsar-io.yaml
      
      * Add comment
      
      * Default is false
      
      * Modify pom file of flume
      
      * Format pom.xml file
      
      * Move pom version to 2.4.0-SNAPSHOT
      37a2fda6
  9. 16 2月, 2019 1 次提交
  10. 13 2月, 2019 1 次提交
    • B
      Add a Pulsar IO MongoDB (#3561) · 06e911d4
      Bruno Bonnin 提交于
      ### Motivation
      
      Provides a builtin MongoDB Connector, in order to ease the storage of JSON formated message in MongoDB. It's a sink connector.
      
      
      ### Modifications
      
      Add a new sub-module in the `pulsar-io`module.
      
      ### Verifying this change
      
      This change added tests and can be verified as follows:
      * deploy the connector with configuration file containing the following fields:
      ```
      configs:
        mongoUri: mongodb://hostname:port
        database: pulsar
        collection: messages
      ```
      * start a mongodb instance
      * send messages in the topic declared when deploying the connector
      * check in MongoDB if the messages have been stored in the collection `messages`
      06e911d4
  11. 01 2月, 2019 1 次提交
  12. 01 1月, 2019 1 次提交
  13. 28 12月, 2018 1 次提交
    • D
      Added Pulsar IO connector for local files (#2869) · 209c4488
      David Kjerrumgaard 提交于
      ### Motivation
      
      Added a Pulsar IO connector for consuming files from the local filesystem
      
      ### Modifications
      
      Added a new module to the pulsar-io module that includes the Pulsar file connector and its associated classes & tests
      
      ### Result
      
      After your change, users will be able to consume files from the local filesystem, and have the contents directly published to a Pulsar topic.
      209c4488
  14. 16 12月, 2018 2 次提交
    • D
      Broke Pulsar-IO::HDFS into 3.x and 2.x versions (#2966) · 2ccf7ff2
      David Kjerrumgaard 提交于
      ### Motivation
      
      A user was attempting to use the existing HDFS connector to connect to a 2.x version of HDFS, but the current connector only supported 3.x version of HDFS. 
      
      ### Modifications
      
      To address this issue, we renamed the current HDFS connector to HDFS3, and created a new 2.x compatible connector named HDFS2. 
      
      The code in both of these are nearly identical with 2 notable exceptions. First and foremost, they both use different versions of the Hadoop-client library. And secondly, the HDFS2 version creates the FSDataOutputStream object directly, whereas the HDFS3 version leverages the FSDataOutputStreamBuilder class for this purpose, as it is the preferred method going forward.
      
      ### Result
      
      There will be be support for connecting to both 2.x and 3.x version of HDFS. However, there MAY BE some library conflicts in the released jar due to the different versions of the same library in the different modules. Hopefully the NAR packaging will address this.
      2ccf7ff2
    • E
      [Pulsar-IO] Add Netty Tcp Source Support (#3179) · 576609b0
      Eren Avsarogullari 提交于
      ### Motivation
      Netty is NIO client server framework by supporting asynchronous event-driven communication and custom protocol implementation.
      Ref: https://netty.io/
      
      This PR proposes Pulsar-IO Netty Source Connector by aiming the Tcp clients. It enables an embedded Tcp Server to listen incoming Tcp messages and writes them to user-defined Pulsar topic.
      
      There are also other potential use-cases(Tcp, Http and Udp messages) for this module as follows:
      - Tcp Client (Pulsar-IO Sink): It can listen Pulsar messages and can write to remote Tcp Server.
      - Http Server and Client (Pulsar-IO Source and Sink)
      - Udp Server and Client (Pulsar-IO Source and Sink)
      
      This is following PR with #3095. Module has been rolled to Pulsar-IO in the light of the previous discussion.
      
      ### Modifications
      1- `NettyTcpServer`: Initializes an embedded Tcp Server to listen incoming Tcp Requests
      2- `NettyTcpServerHandler`: Inbound Channel Handler to handle incoming Tcp Requests
      3- `NettyChannelInitializer`: Channel Initializer to support different types of decoders and handlers
      4- `NettyTcpSource`: A push-based Source to listen Tcp messages and write them to user-defined Pulsar topic
      5- `NettyTcpSourceConfig`: To support user-defined config for both Map and Yaml. 
      6- UT Coverages
      576609b0
  15. 26 11月, 2018 2 次提交
    • S
      [io][docs] introduce annotations for generating connector yaml config files (#2936) · 2c8f0810
      Sijie Guo 提交于
      
      *Motivation*
      
      Currently all io connectors lack example yaml files. Manually write those files is error-prone.
      We need a programmable way that automatically generate example connector yaml files.
      
      *Changes*
      
      - introduce annotations for documenting connector yaml files.
      - provide a generator to generate yaml files
      - provide a shell script to run generator
      - when building io package, generate yaml configs
      
      2c8f0810
    • T
      support mysql binlog sync to pulsar by canal (#2998) · c2c9f33e
      tuteng 提交于
      ### Motivation
      
      support alibaba canal
      https://github.com/alibaba/canal/wiki
      
      ### Modifications
      
      Integrated canal client
      
      ### Result
      
      support binlog sync to pulsar
      use python pulsar-client to consume
      
      ```
      import pulsar
      
      client = pulsar.Client('pulsar://localhost:6650')
      consumer = client.subscribe('my-topic',
                                  subscription_name='my-sub')
      
      while True:
          msg = consumer.receive()
          print("Received message: '%s'" % msg.data())
          consumer.acknowledge(msg)
      
      client.close()
      ```
      
      ```
      output:
      Received message: '[{"data":null,"database":"testdb","es":1542446501000,"id":44,"isDdl":true,"mysqlType":null,"old":null,"sql":"CREATE TABLE `users320` (   `id` int(11) NOT NULL AUTO_INCREMENT,   `name` varchar(50) DEFAULT NULL,   `extra` varchar(50) DEFAULT NULL,   PRIMARY KEY (`id`),   KEY `ix_users_name` (`name`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8","sqlType":null,"table":"users320","ts":1542446501114,"type":"CREATE"}]'
      ```
      c2c9f33e
  16. 11 10月, 2018 1 次提交
  17. 04 10月, 2018 1 次提交
  18. 24 9月, 2018 1 次提交
  19. 21 9月, 2018 1 次提交
  20. 11 9月, 2018 1 次提交
    • D
      Elastic connector (#2546) · 928c3e1a
      David Kjerrumgaard 提交于
      ### Motivation
      
      Added a sink connector that writes JSON documents into ElasticSearch
      
      ### Modifications
      
      Added new pulsar-io module and associated integration tests
      
      ### Result
      
      An ElasticSearch sink connector will be available for use.
      928c3e1a
  21. 07 9月, 2018 1 次提交
    • D
      Added HDFS Sink (#2409) · 261eab12
      David Kjerrumgaard 提交于
      * Added HDFS Sink
      
      * Fixed issues identified during PR review
      
      * Fixed comment
      
      * Added HDFS Container to externalServices
      
      * Ignoring HdfsSink test for now
      
      * Removed HDFS Container to externalServices
      
      * Fixed ASL licensing
      
      * Fixed compile errors
      
      * Added HDFS to SinkType Enum
      261eab12
  22. 06 9月, 2018 1 次提交
  23. 05 9月, 2018 1 次提交
    • J
      Issue 2313: create a JDBC sink connector (#2440) · 79ac46a6
      Jia Zhai 提交于
      ### Motivation
      
      This change is trying to add a basic JDBC sink connector. 
      
      ### Modifications
      
      Add the jdbc module to the pulsar-io sub-module.
      Add unit test and integration test for it.
      
      ### Result
      
      ut and integration test pass.
      
      Master Issue: #2442 
      79ac46a6
  24. 01 9月, 2018 1 次提交
    • B
      Initial SQL documentation and DataGeneratorSource (#2479) · 18f0741b
      Boyang Jerry Peng 提交于
      * Initial SQL documentation and DataGeneratorSource
      
      * adding license header
      
      * adding to sidebar
      
      * improving documentation
      
      * adding to sql getting started
      
      * small fix
      
      * adding data generator to connector bin distribution
      
      * improve sql worker cli
      
      * adding data generator to io pom
      
      * modifying launch args
      18f0741b
  25. 28 8月, 2018 1 次提交
  26. 26 6月, 2018 1 次提交
  27. 08 6月, 2018 1 次提交
  28. 09 5月, 2018 1 次提交
    • L
      Pulsar IO rename (#1747) · 17f74150
      Luc Perkins 提交于
      * begin renaming process
      
      * more class and directory renames
      
      * move Record classes into pulsar-io
      
      * apply rename to Maven configs
      
      * rename java imports
      
      * update versions in maven configs
      
      * add missing imports
      
      * remove Message class from pulsar-io
      
      * add missing import
      
      * add Reflections util import
      
      * add Utils import
      
      * add missing Record import
      
      * supply missing Record imports
      17f74150
  29. 03 5月, 2018 1 次提交
  30. 13 4月, 2018 2 次提交
  31. 11 4月, 2018 2 次提交
  32. 10 4月, 2018 1 次提交
    • S
      Pulsar Connect (#1520) · 027b424f
      Sanjeev Kulkarni 提交于
      * Added Pulsar Connect interfaces that define connectors that push data into pulsar and take data from pulsar
      
      * Added Twitter connector
      
      * Added hbc core version mapping
      
      * Addressed comments
      
      * Fixed build
      
      * Fixed license header
      027b424f
  33. 14 2月, 2018 1 次提交
  34. 29 11月, 2017 1 次提交
  35. 06 10月, 2017 1 次提交
  36. 17 9月, 2017 1 次提交