README.md 9.9 KB
Newer Older
A
Avi Aryan 已提交
1 2 3
[![Build Status](https://travis-ci.org/compose/transporter.svg?branch=master)](https://travis-ci.org/compose/transporter) [![Go Report Card](https://goreportcard.com/badge/github.com/compose/transporter)](https://goreportcard.com/report/github.com/compose/transporter) [![codecov](https://codecov.io/gh/compose/transporter/branch/master/graph/badge.svg)](https://codecov.io/gh/compose/transporter) [![Docker Repository on Quay](https://quay.io/repository/compose/transporter/status "Docker Repository on Quay")](https://quay.io/repository/compose/transporter) [![Gitter](https://img.shields.io/gitter/room/nwjs/nw.js.svg)](https://gitter.im/compose-transporter/Lobby)


4
# ABC
A
Avi Aryan 已提交
5

6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105
1. [Intro](#intro)
2. [Installation](#installation)
  1. [Basic Installation](#basic-installation)
  2. [Using Docker](#using-docker)
3. [Features](#features)


<a name="intro"></a>
## 1. Intro

ABC is a command-line client for appbase.io with nifty features to do data sync from on store to another.

It consists of two parts. 

1. Appbase module
2. Import module

To get the list of all commands supported by ABC, use -

```sh
abc --help
```


<a name="installation"></a>
## 2. Installation

ABC can be installed and used via the traditional `go build` or using a Docker image.


<a name="basic-installation"></a>
### 2.1 Basic installation

You can install ABC by building it locally and then moving the executable to anywhere you like. 

To build it, you require **Go 1.8** insalled on your system. 

```sh
go get github.com/appbaseio-confidential/abc
cd $GOPATH/src/github.com/appbaseio-confidential/abc
go build -tags 'oss' ./cmd/abc/...
./abc --help
```

<a name="using-docker"></a>
### 2.2 Using Docker

```sh
git clone https://github.com/appbaseio-confidential/abc
cd abc
docker build -t abc .
docker volume create --name abc
```

Volume is used to store abc config files across containers.
Now `abc` can be ran through Docker like in the following example which starts google login.  

```sh
docker run -i --rm -v abc:/root abc login google
```

Some more examples

```sh
docker run -i --rm -v abc:/root abc user
docker run -i --rm -v abc:/root abc apps
```


<a name="features"></a>
## 3. Features

ABC's features can be broadly categorized into 2 components. 

1. Appbase features
2. Importer features


### 3.1 Appbase features

Appbase features allows you to control your appbase.io account using ABC. You can see them under the *Appbase* heading in the list of commands.

```sh
APPBASE
  login     login into appbase.io
  user      get user details
  apps      display user apps
  app       display app details
  create    create app
  delete    delete app
```

You can look over help for each of these commands using the `--help` switch. 

```sh
abc login --help
```


### 3.2 Importer features
A
Avi Aryan 已提交
106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132

Transporter allows the user to configure a number of data adaptors as sources or sinks. These can be databases, files or other resources. Data is read from the sources, converted into a message format, and then send down to the sink where the message is converted into a writable format for its destination. The user can also create data transformations in JavaScript which can sit between the source and sink and manipulate or filter the message flow.

Adaptors may be able to track changes as they happen in source data. This "tail" capability allows a Transporter to stay running and keep the sinks in sync.

***BETA Feature***

As of release `v0.4.0`, transporter contains support for being able to resume operations
after being stopped. The feature is disabled by default and can be enabled with the following:

```
source = mongodb({"uri": "mongo://localhost:27017/source_db"})
sink = mongodb({"uri": "mongo://localhost:27017/sink_db"})
t.Config({"log_dir":"/data/transporter"})
  .Source("source", source)
  .Save("sink", sink)
```

When using the above pipeline, all messages will be appended to a commit log and 
successful processing of a message is handled via consumer/sink offset tracking.

Below is a list of each adaptor and its support of the feature:

```
+---------------+-------------+----------------+
|    adaptor    | read resume | write tracking |
+---------------+-------------+----------------+
133
| elasticsearch |             |       X        | 
A
Avi Aryan 已提交
134 135 136 137 138 139 140 141
|     file      |             |       X        | 
|    mongodb    |      X      |       X        | 
|  postgresql   |             |       X        | 
|   rabbitmq    |      X      |                | 
|   rethinkdb   |             |       X        | 
+---------------+-------------+----------------+
```

142
#### Adaptors
A
Avi Aryan 已提交
143 144 145 146 147 148 149 150 151 152

Each adaptor has its own README page with details on configuration and capabilities.

* [elasticsearch](./adaptor/elasticsearch)
* [file](./adaptor/file)
* [mongodb](./adaptor/mongodb)
* [postgresql](./adaptor/postgres)
* [rabbitmq](./adaptor/rabbitmq)
* [rethinkdb](./adaptor/rethinkdb)

153
#### Native Functions
A
Avi Aryan 已提交
154 155 156 157 158 159 160 161 162 163 164

Each native function can be used as part of a `Transform` step in the pipeline.

* [goja](./function/gojajs)
* [omit](./function/omit)
* [otto](./function/ottojs)
* [pick](./function/pick)
* [pretty](./function/pretty)
* [rename](./function/rename)
* [skip](./function/skip)

165
#### Commands
A
Avi Aryan 已提交
166

167
##### init
A
Avi Aryan 已提交
168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205

```
transporter init [source adaptor name] [sink adaptor name]
```

Generates a basic `pipeline.js` file in the current directory.

_Example_
```
$ transporter init mongodb elasticsearch
$ cat pipeline.js
var source = mongodb({
  "uri": "${MONGODB_URI}"
  // "timeout": "30s",
  // "tail": false,
  // "ssl": false,
  // "cacerts": ["/path/to/cert.pem"],
  // "wc": 1,
  // "fsync": false,
  // "bulk": false,
  // "collection_filters": "{}"
})

var sink = elasticsearch({
  "uri": "${ELASTICSEARCH_URI}"
  // "timeout": "10s", // defaults to 30s
  // "aws_access_key": "ABCDEF", // used for signing requests to AWS Elasticsearch service
  // "aws_access_secret": "ABCDEF" // used for signing requests to AWS Elasticsearch service
})

t.Source(source).Save(sink)
// t.Source("source", source).Save("sink", sink)
// t.Source("source", source, "namespace").Save("sink", sink, "namespace")
$
```

Edit the `pipeline.js` file to configure the source and sink nodes and also to set the namespace.

206
##### about
A
Avi Aryan 已提交
207 208 209 210 211 212

`transporter about`

Lists all the adaptors currently available.

_Example_
213

A
Avi Aryan 已提交
214 215 216 217 218 219 220 221 222 223 224 225
```
elasticsearch - an elasticsearch sink adaptor
file - an adaptor that reads / writes files
mongodb - a mongodb adaptor that functions as both a source and a sink
postgres - a postgres adaptor that functions as both a source and a sink
rabbitmq - an adaptor that handles publish/subscribe messaging with RabbitMQ 
rethinkdb - a rethinkdb adaptor that functions as both a source and a sink
```

Giving the name of an adaptor produces more detail, such as the sample configuration.

_Example_
226

A
Avi Aryan 已提交
227 228 229 230 231 232 233 234 235 236 237 238 239
```
transporter about postgres
postgres - a postgres adaptor that functions as both a source and a sink

 Sample configuration:
{
  "uri": "${POSTGRESQL_URI}"
  // "debug": false,
  // "tail": false,
  // "replication_slot": "slot"
}
```

240
##### run
A
Avi Aryan 已提交
241 242 243 244 245 246 247

```
transporter run [-log.level "info"] <application.js>
```

Runs the pipeline script file which has its name given as the final parameter.

248
##### test
A
Avi Aryan 已提交
249 250 251 252 253 254 255 256

```
transporter test [-log.level "info"] <application.js>
```

Evaluates and connects the pipeline, sources and sinks. Establishes connections but does not run.
Prints out the state of connections at the end. Useful for debugging new configurations.

257
##### xlog
A
Avi Aryan 已提交
258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291

The `xlog` command is useful for inspecting the current state of the commit log.
It contains 3 subcommands, `current`, `oldest`, and `offset`, as well as 
a required flag `-log_dir` which should be the path to where the commit log is stored.

***NOTE*** the command should only be run against the commit log when transporter
is not actively running.

```
transporter xlog -log_dir=/path/to/dir current
12345
```

Returns the most recent offset appended to the commit log.

```
transporter xlog -log_dir=/path/to/dir oldest
0
```

Returns the oldest offset in the commit log.

```
transporter xlog -log_dir=/path/to/dir show 0
offset    : 0
timestamp : 2017-05-16 11:00:20 -0400 EDT
mode      : COPY
op        : INSERT
key       : MyCollection
value     : {"_id":{"$oid":"58efd14b60d271d7457b4f24"},"i":0}
```

Prints out the entry stored at the provided offset.

292
##### offset
A
Avi Aryan 已提交
293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336

The `offset` command provides access to current state of each consumer (i.e. sink)
offset. It contains 4 subcommands, `list`, `show`, `mark`, and `delete`, as well as 
a required flag `-log_dir` which should be the path to where the commit log is stored.

```
transporter offset -log_dir=/path/to/dir list
+------+---------+
| SINK | OFFSET  |
+------+---------+
| sink | 1103003 |
+------+---------+
```

Lists all consumers and their associated offset in `log_dir`.

```
transporter offset -log_dir=/path/to/dir show sink
+-------------------+---------+
|     NAMESPACE     | OFFSET  |
+-------------------+---------+
| newCollection     | 1102756 |
| testC             | 1103003 |
| MyCollection      |  999429 |
| anotherCollection | 1002997 |
+-------------------+---------+
```

Prints out each namespace and its associated offset.

```
transporter offset -log_dir=/path/to/dir mark sink 1
OK
```

Rewrites the namespace offset map based on the provided offset.

```
transporter offset -log_dir=/path/to/dir delete sink
OK
```

Removes the consumer (i.e. sink) log directory.

337
##### flags
A
Avi Aryan 已提交
338 339 340 341 342

`-log.level "info"` - sets the logging level. Default is info; can be debug or error.



343
## Building guides
A
Avi Aryan 已提交
344

345 346 347
[macOS](https://github.com/appbaseio-confidential/abc/blob/master/READMEMACOS.md)
[Windows](https://github.com/appbaseio-confidential/abc/blob/master/READMEWINDOWS.md)
[Vagrant](https://github.com/appbaseio-confidential/abc/blob/master/READMEVAGRANT.md)
A
Avi Aryan 已提交
348 349


350
## ABC Resources
A
Avi Aryan 已提交
351

352
* [ABC Wiki](https://github.com/appbaseio-confidential/abc/wiki)
A
Avi Aryan 已提交
353 354


355 356 357
## Contributing to ABC

Want to help out with ABC? Great! There are instructions to get you
A
Avi Aryan 已提交
358 359 360
started [here](CONTRIBUTING.md).


361 362 363 364
## Licensing

ABC is licensed under the New BSD License. See [LICENSE](LICENSE) for full license text.