未验证 提交 69d45306 编写于 作者: W Wu Tao 提交者: GitHub

chore: fix stale urls and remove unmaintained files (#620)

上级 be2c6c2e
......@@ -8,6 +8,6 @@ about: General question that isn't answered in docs
Before asking a question, make sure you have:
- Searched open and closed [GitHub issues](https://github.com/XiaoMi/pegasus/issues)
- Searched open and closed [GitHub issues](https://github.com/apache/incubator-pegasus/issues)
- Read the documentation:
- [Pegasus Doc](https://pegasus.apache.org)
#!/bin/bash
set -e
SCRIPT_DIR=$(dirname "${BASH_SOURCE[0]}")
ROOT=$(
cd "$(dirname "${SCRIPT_DIR}")" || exit 1
pwd
)
SERVER_PKG_NAME=$(cat "${ROOT}"/PACKAGE)
###
# configurable variables
IMAGE_NAME=pegasus:latest
###
echo "Building image ${IMAGE_NAME}"
cd "${ROOT}" || exit 1
if [[ -z ${SERVER_PKG_NAME} ]]; then
echo "SERVER_PKG_NAME is empty"
exit 1
fi
if [[ ! -f "${ROOT}"/${SERVER_PKG_NAME}.tar.gz ]]; then
echo "Failed to find package ${SERVER_PKG_NAME}.tar.gz in ${ROOT}"
exit 1
else
echo "Found package ${ROOT}/${SERVER_PKG_NAME}.tar.gz"
fi
cd "${ROOT}" || exit 1
mkdir -p ${IMAGE_NAME}
cp -f docker/dev/linux/entrypoint.sh ${IMAGE_NAME}/entrypoint.sh
cp -f docker/dev/linux/Dockerfile ${IMAGE_NAME}/Dockerfile
sed -i 's/@SERVER_PKG_NAME@/'"${SERVER_PKG_NAME}"'/' ${IMAGE_NAME}/Dockerfile
cp ${SERVER_PKG_NAME}.tar.gz ${IMAGE_NAME}
cd ${IMAGE_NAME} || exit 1
docker build --build-arg SERVER_PKG_NAME="${SERVER_PKG_NAME}" -t ${IMAGE_NAME} .
cd "${ROOT}" || exit 1
docker images
# clean up
rm -r ${IMAGE_NAME}
#!/usr/bin/env bash
cd "$( dirname "${BASH_SOURCE[0]}" )" || exit 1
source cluster_args.sh
if ! [[ -d "${DOCKER_DIR}" ]]; then
echo "Cleared ${DOCKER_DIR} already"
exit 0
fi
cd "${DOCKER_DIR}" || exit 1
pwd
docker-compose kill
docker-compose rm -f -v
cd "${ROOT}" || exit 1
./run.sh clear_zk
sudo rm -rf "${DOCKER_DIR}"
#!/usr/bin/env bash
# Configure the following variables to customize the docker cluster. #
export PARTITION_COUNT=8
export APP_NAME=temp
# The ip prefix for each nodes.
# Meta-x's ip address is 172.21.0.1{x}:34601.
# Replica-x's ip address is 172.21.0.2{x}:34801
export NODE_IP_PREFIX=172.21.0
export CLUSTER_NAME=onebox2
export IMAGE_NAME=pegasus:latest
# allow_non_idempotent_write = true
# for jepsen test this option must be enabled.
export IDEMPOTENT=true
# Config End #
##############
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
ROOT=$(dirname "${SCRIPT_DIR}")
LOCAL_IP=$("${ROOT}"/scripts/get_local_ip)
export SCRIPT_DIR
export ROOT
export LOCAL_IP
export DOCKER_DIR=${ROOT}/${CLUSTER_NAME}-docker # Where docker onebox resides.
export META_COUNT=2 # Number of meta instances.
export REPLICA_COUNT=5 # Number of replica instances.
# This is a docker with all stuff built after compilation. It is a
# large image that's only used for testing.
#
# ```sh
# docker build -t pegasus_onebox .
# docker run -it pegasus_onebox bash
# ```
FROM centos:7
MAINTAINER Wu Tao <wutao1@xiaomi.com>
RUN yum -y install gcc gcc-c++ automake autoconf libtool make cmake git file wget unzip python-devel which && \
yum -y install openssl-devel boost-devel libaio-devel snappy-devel bzip2-devel zlib zlib-devel patch
RUN git clone --recursive https://github.com/XiaoMi/pegasus.git /pegasus
RUN cd /pegasus && \
./run.sh build -c
RUN yum -y install jre nmap-ncat.x86_64
CMD cd /pegasus
FROM jacobmbr/ubuntu-jepsen:v0.1.0
# Usage:
#
# ./run.sh build -c
# ./run.sh pack_server
# mv pegasus-server-{YOUR_VERSION}.tar.gz docker/dev/jepsen/
# cd docker/dev/jepsen/
# docker build --build-arg SERVER_PKG_NAME=pegasus-server-{YOUR_VERSION} -t pegasus:latest .
# Install Jepsen dependencies
RUN sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/' /etc/apt/sources.list; \
rm /etc/apt/apt.conf.d/docker-clean && apt-get update -y
RUN apt-get install -y openssh-server \
curl faketime iproute2 iptables iputils-ping libzip4 \
logrotate man man-db net-tools ntpdate psmisc python rsyslog \
sudo unzip vim wget apt-transport-https \
&& apt-get remove -y --purge --auto-remove systemd
ARG SERVER_PKG_NAME
COPY ./$SERVER_PKG_NAME.tar.gz /
RUN tar xvf /$SERVER_PKG_NAME.tar.gz; \
mv $SERVER_PKG_NAME pegasus; \
rm $SERVER_PKG_NAME.tar.gz
COPY ./entrypoint.sh /
RUN chmod +x /entrypoint.sh
ENV LD_LIBRARY_PATH=/pegasus/bin
ENTRYPOINT ["/entrypoint.sh"]
#!/bin/bash
/usr/sbin/sshd -D &
/pegasus/bin/pegasus_server /pegasus/bin/config.ini -app_list "$1"
FROM ubuntu:18.04
# This is a docker image for any usage once you have a pre-built
# binary of Pegasus.
#
# Usage:
#
# ./run.sh build -c
# ./run.sh pack_server
# mv pegasus-server-{YOUR_VERSION}.tar.gz docker/dev/linux/
# cd docker/dev/linux/
# docker build --build-arg SERVER_PKG_NAME=pegasus-server-{YOUR_VERSION} -t pegasus:latest .
#
# Or simply run docker/build_docker.sh to build image named pegasus:latest.
#
ARG SERVER_PKG_NAME
# Install libunwind
RUN sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/' /etc/apt/sources.list; \
rm /etc/apt/apt.conf.d/docker-clean && apt-get update -y; \
apt-get install -y libunwind-dev libgssapi-krb5-2; \
rm -rf /var/lib/apt/lists/*
COPY ./$SERVER_PKG_NAME.tar.gz /
RUN tar xvf /$SERVER_PKG_NAME.tar.gz; \
mv $SERVER_PKG_NAME pegasus; \
rm $SERVER_PKG_NAME.tar.gz
COPY ./entrypoint.sh /
RUN chmod +x /entrypoint.sh
ENV LD_LIBRARY_PATH=/pegasus/bin
ENTRYPOINT ["/entrypoint.sh"]
#!/bin/bash
/pegasus/bin/pegasus_server /pegasus/bin/config.ini -app_list "$1"
FROM ubuntu:16.04
MAINTAINER WU TAO <wutao1@xiaomi.com>
RUN sed -i 's/archive.ubuntu.com/mirrors.aliyun.com/' /etc/apt/sources.list; \
apt-get update -y; \
apt-get -y install build-essential cmake libboost-system-dev libboost-filesystem-dev libboost-regex-dev \
libaio-dev libsnappy-dev libbz2-dev libtool \
zlib1g zlib1g.dev patch git curl zip automake libssl-dev;
RUN git clone --recursive https://github.com/XiaoMi/pegasus.git /pegasus; \
cd /pegasus; \
./run.sh build; \
./run.sh pack_tools; \
cp -r pegasus-tools-* /; cd /; rm -rf /pegasus \
rm pegasus-tools-*.tar.gz;
RUN apt-get -y install default-jre nmap netcat
version: '3.3'
networks:
frontend:
ipam:
config:
- subnet: @NODE_IP_PREFIX@.0/24
services:
meta1:
image: @IMAGE_NAME@
ports:
- "34601"
volumes:
- ./config.ini:/pegasus/bin/config.ini:ro
- ./meta1/data:/pegasus/data
command:
- meta
privileged: true
networks:
frontend:
ipv4_address: @NODE_IP_PREFIX@.11
restart: on-failure
meta2:
image: @IMAGE_NAME@
ports:
- "34601"
volumes:
- ./config.ini:/pegasus/bin/config.ini:ro
- ./meta2/data:/pegasus/data
command:
- meta
privileged: true
networks:
frontend:
ipv4_address: @NODE_IP_PREFIX@.12
restart: on-failure
replica1:
image: @IMAGE_NAME@
ports:
- "34801"
volumes:
- ./config.ini:/pegasus/bin/config.ini:ro
- ./replica1/data:/pegasus/data
- ./replica1/slog:/pegasus/slog
command:
- replica
privileged: true
networks:
frontend:
ipv4_address: @NODE_IP_PREFIX@.21
restart: on-failure
replica2:
image: @IMAGE_NAME@
ports:
- "34801"
volumes:
- ./config.ini:/pegasus/bin/config.ini:ro
- ./replica2/data:/pegasus/data
- ./replica2/slog:/pegasus/slog
command:
- replica
privileged: true
networks:
frontend:
ipv4_address: @NODE_IP_PREFIX@.22
restart: on-failure
replica3:
image: @IMAGE_NAME@
ports:
- "34801"
volumes:
- ./config.ini:/pegasus/bin/config.ini:ro
- ./replica3/data:/pegasus/data
- ./replica3/slog:/pegasus/slog
command:
- replica
privileged: true
networks:
frontend:
ipv4_address: @NODE_IP_PREFIX@.23
restart: on-failure
replica4:
image: @IMAGE_NAME@
ports:
- "34801"
volumes:
- ./config.ini:/pegasus/bin/config.ini:ro
- ./replica4/data:/pegasus/data
- ./replica4/slog:/pegasus/slog
command:
- replica
privileged: true
networks:
frontend:
ipv4_address: @NODE_IP_PREFIX@.24
restart: on-failure
replica5:
image: @IMAGE_NAME@
ports:
- "34801"
volumes:
- ./config.ini:/pegasus/bin/config.ini:ro
- ./replica5/data:/pegasus/data
- ./replica5/slog:/pegasus/slog
command:
- replica
privileged: true
networks:
frontend:
ipv4_address: @NODE_IP_PREFIX@.25
restart: on-failure
#!/bin/bash
cd "$( dirname "${BASH_SOURCE[0]}" )" || exit 1
source cluster_args.sh
function print_nodes()
{
echo ""
for i in $(seq "${META_COUNT}"); do
echo "META${i}: ${NODE_IP_PREFIX}.1$((i)):34601"
done
for i in $(seq "${REPLICA_COUNT}"); do
echo "REPLICA${i}: ${NODE_IP_PREFIX}.2$((i)):34801"
done
cd "${DOCKER_DIR}" || exit 1
echo ""
docker-compose ps
echo ""
cd - || exit 1
}
if ! [[ -x "$(command -v docker-compose)" ]]; then
echo 'ERROR: docker-compose is not installed.' >&2
echo 'See this document for installation manual:' >&2
echo ' https://docs.docker.com/compose/install' >&2
exit 1
fi
if [[ -d "${DOCKER_DIR}" ]]; then
echo "ERROR: ${DOCKER_DIR} already exists, please remove it first" >&2
print_nodes
exit 1
fi
mkdir -p "${DOCKER_DIR}"
cp -f "${ROOT}"/src/server/config.ini "${DOCKER_DIR}/config.ini"
sed -i 's/@META_PORT@/34601/' "${DOCKER_DIR}/config.ini"
sed -i 's/@REPLICA_PORT@/34801/' "${DOCKER_DIR}/config.ini"
sed -i 's/%{cluster.name}/'"${CLUSTER_NAME}"'/g' "${DOCKER_DIR}/config.ini"
sed -i 's/%{app.dir}/\/pegasus\/data/g' "${DOCKER_DIR}/config.ini"
sed -i 's/%{slog.dir}/\/pegasus\/slog/g' "${DOCKER_DIR}/config.ini"
sed -i 's/%{data.dirs}//g' "${DOCKER_DIR}/config.ini"
sed -i 's@%{home.dir}@'"/pegasus"'@g' "${DOCKER_DIR}/config.ini"
sed -i "s/allow_non_idempotent_write = false/allow_non_idempotent_write = ${IDEMPOTENT}/" "${DOCKER_DIR}/config.ini"
for i in $(seq "${META_COUNT}"); do
meta_port=34601
meta_ip=${NODE_IP_PREFIX}.1$((i))
if [ "${i}" -eq 1 ]; then
meta_list="${meta_ip}:$meta_port"
else
meta_list="$meta_list,${meta_ip}:$meta_port"
fi
done
sed -i 's/%{meta.server.list}/'"$meta_list"'/g' "${DOCKER_DIR}/config.ini"
sed -i 's/%{zk.server.list}/'"${LOCAL_IP}"':22181/g' "${DOCKER_DIR}/config.ini"
sed -i 's/app_name = .*$/app_name = '"$APP_NAME"'/' "${DOCKER_DIR}/config.ini"
sed -i 's/partition_count = .*$/partition_count = '"$PARTITION_COUNT"'/' "${DOCKER_DIR}/config.ini"
cp -f "${SCRIPT_DIR}"/docker-compose.yml "${DOCKER_DIR}"
sed -i 's/@NODE_IP_PREFIX@/'"${NODE_IP_PREFIX}"'/' "${DOCKER_DIR}"/docker-compose.yml
sed -i 's/@IMAGE_NAME@/'"${IMAGE_NAME}"'/' "${DOCKER_DIR}"/docker-compose.yml
echo "${DOCKER_DIR} is ready"
print_nodes
#!/usr/bin/env bash
cd "$( dirname "${BASH_SOURCE[0]}" )" || exit 1
source cluster_args.sh
cd "${ROOT}" || exit 1
./docker/clear_onebox.sh
./docker/prepare.sh
./run.sh start_zk
cd "${DOCKER_DIR}" || exit 1
pwd
docker-compose up -d
Notices For Client Library Development
==========
Currently C++ and Java are supported to visit Pegasus cluster. You may want to refer to [C++ documentations](https://github.com/XiaoMi/pegasus/wiki/Cpp%E5%AE%A2%E6%88%B7%E7%AB%AF%E6%96%87%E6%A1%A3) or [Java documentation](https://github.com/XiaoMi/pegasus/wiki/Java%E5%AE%A2%E6%88%B7%E7%AB%AF%E6%96%87%E6%A1%A3) for details.
Here we give some notices on the client library, which may be helpful to development of other language bindings.
## message protocol
### request to server
Message format sent to some single server of Pegasus Cluster:
`
(A special header of 48 bytes) + (body)
`
#### header
The 48-byte header consists of:
| bytes | type | comments
|-------| -----|--------- |
| 0~3 | "THFT" | (header_type)
| 4~7 | 32bit int | header version
| 8~11 | 32bit int | header_length
| 12~15 | 32bit int | header_crc32
| 16~19 | 32bit int | body_length
| 20~23 | 32bit int | body_crc32
| 24~27 | 32bit int | app_id (rDSN related concept)
| 28~31 | 32bit int | partition_index (rDSN related concept)
| 32~35 | 32bit int | client_timeout
| 36~39 | 32bit int | thread_hash (rDSN related concept)
| 40~47 | 64bit long | partition_hash (rDSN related concept)
Some notes on the above "header":
* all ints/long in the header are in network order.
* if send request to meta server, the app\_id & partition\_index should set to 0; if request is to replica server, the fields should be set to the target replica.
* pegasus server may use client_timeout to do some optimization. Say, a server may simply discard a request if it has expired when server receives it
* thread\_hash = app\_id * 7919 + partition\_index, it is used for server to decide in which thread to queue the request.
* partition_hash should set to 0, it's only a useful field for RPC client of rDSN framework.
#### body
the body is a standard thrift struct in binary protocol:
TMessageBegin + args + TMessageEnd.
You should write a thrift "TMessage" in TMessageBegin, the structure of TMessage is:
* name: the RPC name (please refer to Java client for detail)
* type: TMessage.CALL
* seqid: the seqid int
### response from server
response got from pegasus server:
`
total_response_length(4 bytes) + error_code_thrift_struct + response_body
`
Some notes on the above response:
* **error_code_thrift_struct** is an error_code struct in thrift binary protocol, usually it indicates some error of service status than the response of some specific rpc call. For example, for a meta server, this error may indicate "the meta server is not leader"; for a read/write request to replica server, this error may indicate that the replica server is not a primary or don't serve the partition"
* **response_body** is a standard thrift rpc response of the rpc call, with the format as follows:
* TMessageBegin: rpc_name, TMessage.T_REPLY, seqid_integer
* response_args
* TMessageEnd
## write/read request process
You can refer to [TableHandler.java](https://github.com/XiaoMi/pegasus-java-client/blob/thrift-0.11.0-inlined/src/main/java/com/xiaomi/infra/pegasus/rpc/async/TableHandler.java) for the detailed RPC process in write/read request RPCs.
## how to generate code in thrift
There are [3 IDL files](https://github.com/XiaoMi/pegasus-java-client/tree/thrift-0.11.0-inlined/idl) for RPC client:
* base.thrift: a placeholder for rDSN specific structures(blob, error\_code, task\_code, RPC_address, gpid), you may use thrift to generate a sketch, and implement the details all by yourself.
* replication.thrift: messages and RPCs used for communicate with meta server. Using generated code is ok.
* rrdb.thrift: messages and RPCs used for communicate with replica server. Using generated code is ok.
Due to some history reasons, RPC names defined in the IDL can't be recognized by server right now. A proper name should be set manually, please refer to [operators](https://github.com/XiaoMi/pegasus-java-client/tree/thrift-0.11.0-inlined/src/main/java/com/xiaomi/infra/pegasus/operator) for details. Besides, you may also need to refer
to [base](https://github.com/XiaoMi/pegasus-java-client/tree/thrift-0.11.0-inlined/src/main/java/com/xiaomi/infra/pegasus/base) for how to implement serialization for rDSN specific structures.
Installing Pegasus
===========
Currently Pegasus can only be installed from source. Binary package and docker image is coming soon.
## Prerequisites and Requirements
### Hardware
Pegasus can be deployed in standalone mode which runs all the pegasus jobs as different processes on a single host, or in distributed mode which runs different jobs on different hosts. For distributed mode, more than one hosts are need:
* One or more hosts to run zookeeper. We need zookeeper to make the cluster metadata persistence. For a fault tolerant zookeeper cluster, at least 3 are necessary.
* One or more hosts to run Pegasus meta-server. At least 2 are needed for fault tolerance. You may share hosts between meta-server and zookeeper.
* At least 3 hosts to run Pegasus replica-server.
### Operating System requirements
* Linux: CentOS 7, Ubuntu 14.04(Trusty)
* macOS: not supported
* Microsoft Windows: not supported
## Build Pegasus from source
Please notice that Pegasus can not be built until the following packages meet the version requirements:
* Compiler: the gcc version must >= 4.8 && < 8. If you use other compilers, please make sure that C++14 is supported.
* CMake: must >= 3.5.2
* Boost: must >= 1.58
#### Install the dependencies
For Ubuntu:
```
sudo apt-get install build-essential cmake libboost-all-dev libaio-dev libsnappy-dev libbz2-dev libtool libgflags-dev zlib1g zlib1g.dev patch unzip automake libssl-dev liblz4-dev libzstd-dev
```
To start onebox:
```
sudo apt-get install default-jre netcat
```
For CentOS:
```
yum -y install cmake boost-devel libaio-devel snappy-devel bzip2-devel zlib-devel lz4-devel automake libtool patch unzip openssl-devel python-devel libzstd-devel
```
Zstd may not available in the default EPEL. To install the EPEL repository on CentOS-7/6, simply type:
```
yum install epel-release
```
To start onebox:
```
yum install java-1.8.0-openjdk nmap-ncat.x86_64
```
Please make sure you install the proper version of GCC, CMake and Boost.
#### Build
1. clone Pegasus and its all subprojects:
```
git clone https://github.com/xiaomi/pegasus.git --recursive
```
2. build
```
cd pegasus && ./run.sh build
```
if Boost is installed at some custom path , please tell the build script the install path of Boost:
```
./run.sh build -b /your/boost/installation/path
```
Generally, the build process of Pegasus consists of 4 parts:
1. Download the thirdparty dependencies to "rdsn/thirdparty" and build them, and the output headers/libraries/binaries are installed in "rdsn/thirdparty/output".
2. build the rdsn project, which is a dependency of our KV storage. The output headers/libraries/binaries are installed in "rdsn/builder/output". The build script will create a symbolic link "DSN_ROOT" in the pegasus project root.
3. build rocksdb in "rocksdb" dir, which is modified from [facebook/rocksdb](https://github.com/facebook/rocksdb)
4. build pegasus's KV-layer in "src".
**Please make sure the thirdparty are successfully downloaded and built before subsequent parts**.
#### Common Build Problems
* fatal error: <some_thirdparty_header_files>.h: No such file or directory
This means that Third Party Building did not complete correctly and did not generate the header files.
Rerun `./run.sh build` with `--clear_thirdparty`.
## Run in standalone mode
You can play with pegasus with a **onebox** cluster:
```
./run.sh start_onebox
```
When running onebox cluster, all pegasus jobs are running in local host:
* zookeeper: start a zookeeper process with port 22181
* meta-server: start several meta-server processes
* replica-server: start several replica-server processes
You can also check the state of onebox cluster:
```
./run.sh list_onebox
```
Stop the onebox cluster:
```
./run.sh stop_onebox
```
Clean the onebox cluster:
```
./run.sh clear_onebox
```
## Interactive shell
Pegasus provides a shell tool to interact with the cluster. To start the shell, run:
```
./run.sh shell
```
or specify different cluster meta-server address list:
```
./run.sh shell --cluster 127.0.0.1:34601,127.0.0.1:34602
```
Using the shell, you can do lots of things including:
* get cluster information and statistics
* create/drop/list tables (we name it app)
* get/set/del/scan data
* migrate replicas from node to node
* send remote commands to server
* dump mutation log and sstable files
## Running test
Run the unit test by:
```
./run.sh test
```
or run the killing test by:
```
./run.sh start_kill_test
```
## Benchmark
You can start a benchmark client quickly (needs onebox cluster already started):
```
./run.sh bench
```
or start a series of benchmark tests:
```
./scripts/pegasus_bench_run.sh
```
Attention: this bench tool depends on gflags, so it would not work if gflags library
is not installed when building Pegasus.
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册