[github-release]: https://github.com/apache/incubator-pegasus/releases [PacificA]: https://www.microsoft.com/en-us/research/publication/pacifica-replication-in-log-based-distributed-storage-systems/ [pegasus-rocksdb]: https://github.com/xiaomi/pegasus-rocksdb [facebook-rocksdb]: https://github.com/facebook/rocksdb [hbase]: https://hbase.apache.org/ [website]: https://pegasus.apache.org ![pegasus-logo](docs/media-img/pegasus-logo.png) [![Build Status](https://travis-ci.org/apache/incubator-pegasus.svg?branch=master)](https://travis-ci.org/apache/incubator-pegasus) [![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html) [![Releases](https://img.shields.io/github/release/apache/incubator-pegasus.svg)][github-release] **Note**: The `master` branch may be in an *unstable or even broken state* during development. Please use [releases][github-release] instead of the `master` branch in order to get stable binaries. Apache Pegasus is a distributed key-value storage system which is designed to be: - **horizontally scalable**: distributed using hash-based partitioning - **strongly consistent**: ensured by [PacificA][PacificA] consensus protocol - **high-performance**: using [RocksDB][pegasus-rocksdb] as underlying storage engine - **simple**: well-defined, easy-to-use APIs ## Background Pegasus targets to fill the gap between Redis and [HBase][hbase]. As the former is in-memory, low latency, but does not provide a strong-consistency guarantee. And unlike the latter, Pegasus is entirely written in C++ and its write-path relies merely on the local filesystem. Apart from the performance requirements, we also need a storage system to ensure multiple-level data safety and support fast data migration between data centers, automatic load balancing, and online partition split. ## Features - **Persistence of data**: Each write is replicated three-way to different ReplicaServers before responding to the client. Using PacificA protocol, Pegasus has the ability for strong consistent replication and membership changes. - **Automatic load balancing over ReplicaServers**: Load balancing is a builtin function of MetaServer, which manages the distribution of replicas. When the cluster is in an inbalance state, the administrator can invoke a simple rebalance command that automatically schedules the replica migration. - **Cold Backup**: Pegasus supports an extensible backup and restore mechanism to ensure data safety. The location of snapshot could be a distributed filesystem like HDFS or local filesystem. The snapshot storing in the filesystem can be further used for analysis based on [pegasus-spark](https://github.com/pegasus-kv/pegasus-spark). - **Eventually-consistent intra-datacenter replication**: This is a feature we called *duplication*. It allows a change made in the local cluster accesible after a short time period by the remote cluster. It help achieving higher availability of your service and gaining better performance by accessing only local cluster. ## To start using Pegasus See our documentation on [Pegasus Website][website]. ## Client drivers Pegasus has support for serveral languages: - [Java](https://github.com/xiaomi/pegasus-java-client) - [C++](https://github.com/apache/incubator-pegasus/blob/master/src/include/pegasus/client.h) - [Go](https://github.com/xiaomi/pegasus-go-client) - [Python](https://github.com/xiaomi/pegasus-python-client) - [Node.js](https://github.com/xiaomi/pegasus-nodejs-client) - [Scala](https://github.com/xiaomi/pegasus-scala-client) ## Contact us - Send mails to Apache Pegasus dev mailing list: dev@pegasus.apache.org. This is the place where topics around development, community, and problems are officially discussed. Please remember to subsribe the mail list via dev-subscribe@pegasus.apache.org. - Github Issues: submit an issue when you have any idea to improve Pegasus, and when you encountered some bugs or problems. ## Related Projects Submodules: - [rDSN](https://github.com/xiaomi/rdsn) Test tools: - [Java YCSB](https://github.com/xiaomi/pegasus-YCSB) Data import/export tools: - [DataX](https://github.com/xiaomi/pegasus-datax) ## License Copyright 2020 The Apache Software Foundation. Licensed under the Apache License, Version 2.0: