提交 26aa614e 编写于 作者: S Siddharth Kothari

adds docs for kafka adaptor, fixes #79

上级 5cbfdd99
# Import command # Import command
Import command can be used to import data from any supported database source into appbase.io/ES cluster. Import command can be used to import data from any supported database source into appbase.io/ES cluster.
It goes like - It goes like -
``` ```
abc import --src_uri {URI} --src_type {DBType} --tail [URI|Appname] abc import --src_uri {URI} --src_type {DBType} --tail [URI|Appname]
...@@ -28,7 +28,7 @@ At the time of writing, the list of parameters supported looks like - ...@@ -28,7 +28,7 @@ At the time of writing, the list of parameters supported looks like -
--typename=mytype [csv] typeName to use --typename=mytype [csv] typeName to use
``` ```
Note that you only need to set the parameters that are required for the source database type. For example, you don't set `replication_slot` when taking CSV as the source. Note that you only need to set the parameters that are required for the source database type. For example, you don't set `replication_slot` when taking CSV as the source.
**Note** - Help for [transform_file](../importer/transform_file.md) is available here. **Note** - Help for [transform_file](../importer/transform_file.md) is available here.
...@@ -61,6 +61,11 @@ We can even use the app's name as URI once we are logged in. ...@@ -61,6 +61,11 @@ We can even use the app's name as URI once we are logged in.
abc import --src_type=elasticsearch --src_uri=APPNAME2 APPNAME abc import --src_type=elasticsearch --src_uri=APPNAME2 APPNAME
``` ```
### Kafka
```sh
abc import --src_type=kafka --src_uri="kafka://USER:PASS@HOST:PORT/TOPIC1,TOPIC2" "https://USER:PASS@scalr.api.appbase.io/APPNAME"
```
### MongoDB ### MongoDB
...@@ -75,7 +80,7 @@ abc import --src_type=mongodb -t --src_uri="mongodb://USER:PASS@HOST:PORT/DB" "h ...@@ -75,7 +80,7 @@ abc import --src_type=mongodb -t --src_uri="mongodb://USER:PASS@HOST:PORT/DB" "h
abc import --src_type=mssql --src_uri="sqlserver://USER:PASSWORD@SERVER:PORT?database=DBNAME" "https://USER:PASS@scalr.api.appbase.io/APPNAME" abc import --src_type=mssql --src_uri="sqlserver://USER:PASSWORD@SERVER:PORT?database=DBNAME" "https://USER:PASS@scalr.api.appbase.io/APPNAME"
``` ```
For more source URL patterns, see [go-mssqldb](https://github.com/denisenkom/go-mssqldb#connection-parameters-and-dsn)'s GitHub page. For more source URL patterns, see [go-mssqldb](https://github.com/denisenkom/go-mssqldb#connection-parameters-and-dsn)'s GitHub page.
### MySQL ### MySQL
...@@ -84,7 +89,7 @@ For more source URL patterns, see [go-mssqldb](https://github.com/denisenkom/go- ...@@ -84,7 +89,7 @@ For more source URL patterns, see [go-mssqldb](https://github.com/denisenkom/go-
abc import --src_type=mysql --src_uri="USER:PASS@tcp(HOST:PORT)/DBNAME" "https://USER:PASS@scalr.api.appbase.io/APPNAME" abc import --src_type=mysql --src_uri="USER:PASS@tcp(HOST:PORT)/DBNAME" "https://USER:PASS@scalr.api.appbase.io/APPNAME"
``` ```
For more source URL patterns, see [go-sql-driver/mysql](https://github.com/go-sql-driver/mysql#examples)'s GitHub page. For more source URL patterns, see [go-sql-driver/mysql](https://github.com/go-sql-driver/mysql#examples)'s GitHub page.
### Postgres ### Postgres
...@@ -99,7 +104,7 @@ abc import --src_type=postgres -t --replication_slot="standby_replication_slot" ...@@ -99,7 +104,7 @@ abc import --src_type=postgres -t --replication_slot="standby_replication_slot"
abc import --config=test.env abc import --config=test.env
``` ```
File extension doesn't matter. File extension doesn't matter.
The file `test.env` should be an INI/ENV like file with key value pair containing the values of attributes required for importing. The file `test.env` should be an INI/ENV like file with key value pair containing the values of attributes required for importing.
Example of a test.env file is -- Example of a test.env file is --
...@@ -112,5 +117,4 @@ dest_type=elasticsearch ...@@ -112,5 +117,4 @@ dest_type=elasticsearch
dest_uri=https://USER:PASS@scalr.api.appbase.io/APPNAME dest_uri=https://USER:PASS@scalr.api.appbase.io/APPNAME
``` ```
Note that the key names are same as what we have in `import` parameters. Note that the key names are same as what we have in `import` parameters.
# Kafka
The [Kafka](https://kafka.apache.org/) adaptor is capable of tailing in realtime one or more topics.
Here is how a configuration file looks like:
```ini
src_type=kafka
src_uri=kafka://user:pass@SERVER:PORT/TOPIC1,TOPIC2
tail=false
dest_type=elasticsearch
dest_uri=APP_NAME
# dest_uri=https://USER:PASSWORD@SERVER/INDEX
```
The equivalent CLI command looks like:
```sh
abc import --src_type=kafka --src_uri="kafka://localhost:9200/newtopic" myAppbaseApp
```
If no topic is specified in the `src_uri` path either in the config or in the CLI switch, data from all topics will be tailed.
**Note:** `myAppbaseApp` should already exist. Or you can create a new app with the [`abc create`](https://github.com/appbaseio/abc/blob/dev/docs/appbase/create.md) command and use that.
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册