未验证 提交 75a2cfe6 编写于 作者: A Andy McCurdy 提交者: GitHub

Dockerize (#1365)

Provide a docker development and testing environment

* CI (Travis) now runs tests via the same docker environment that is available to developers.
* A simple Makefile has been added to make getting started easier.
    * `make dev` will standup the development environment.
    * `make test` will standup the development environment and also run the test suite.
    * `make clean` will remove the development environment.
[run]
source = redis
**/__pycache__
**/*.pyc
.tox
......@@ -11,3 +11,6 @@ vagrant/.vagrant
.eggs
.idea
.coverage
env
venv
coverage.xml
language: python
cache: pip
matrix:
include:
- env: TOXENV=flake8
- python: 2.7
env: TOXENV=py27-plain
- python: 2.7
env: TOXENV=py27-hiredis
- python: 3.5
env: TOXENV=py35-plain
- python: 3.5
env: TOXENV=py35-hiredis
- python: 3.6
env: TOXENV=py36-plain
- python: 3.6
env: TOXENV=py36-hiredis
- python: 3.7
env: TOXENV=py37-plain
- python: 3.7
env: TOXENV=py37-hiredis
- python: 3.8
env: TOXENV=py38-plain
- python: 3.8
env: TOXENV=py38-hiredis
- python: pypy
env: TOXENV=pypy-plain
- python: pypy
env: TOXENV=pypy-hiredis
- python: pypy3
env: TOXENV=pypy3-plain
- python: pypy3
env: TOXENV=pypy3-hiredis
env:
- DOCKER_COMPOSE_VERSION=1.26.2
before_install:
- wget https://github.com/antirez/redis/archive/6.0.5.tar.gz && mkdir redis_install && tar -xvzf 6.0.5.tar.gz -C redis_install && cd redis_install/redis-6.0.5 && make && src/redis-server --daemonize yes && cd ../..
- redis-cli info
install:
- pip install codecov tox
- sudo rm /usr/local/bin/docker-compose
- curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose
- chmod +x docker-compose
- sudo mv docker-compose /usr/local/bin
services:
- docker
script:
- tox
after_success:
- "if [[ $TOXENV != 'flake8' ]]; then codecov; fi"
- make test
* (in development)
* Provide a development and testing environment via docker. Thanks
@abrookins. #1365
* 3.5.3 (June 1, 2020)
* Restore try/except clauses to __del__ methods. These will be removed
in 4.0 when more explicit resource management if enforced. #1339
......
Contributing
============
Introduction
------------
First off, thank you for considering contributing to redis-py. We value community contributions!
Contributions We Need
----------------------
You may already know what you want to contribute -- a fix for a bug you encountered, or a new feature your team wants to use.
If you don't know what to contribute, keep an open mind! Improving documentation, bug triaging, and writing tutorials are all examples of helpful contributions that mean less work for you.
Your First Contribution
-----------------------
Unsure where to begin contributing? You can start by looking through `help-wanted issues <https://github.com/andymccurdy/redis-py/issues?q=is%3Aopen+is%3Aissue+label%3ahelp-wanted>`_.
Never contributed to open source before? Here are a couple of friendly tutorials:
- http://makeapullrequest.com/
- http://www.firsttimersonly.com/
Getting Started
---------------
Here's how to get started with your code contribution:
1. Create your own fork of redis-py
2. Do the changes in your fork
3. If you need a development environment, run ``make dev``
4. While developing, make sure the tests pass by running ``make test``
5. If you like the change and think the project could use it, send a pull request
The Development Environment
---------------------------
Running ``make dev`` will create a Docker-based development environment that starts the following containers:
* A master Redis node
* A slave Redis node
* Three sentinel Redis nodes
* A test container
The slave is a replica of the master node, using the `leader-follower replication <https://redis.io/topics/replication>`_ feature.
The sentinels monitor the master node in a `sentinel high-availability configuration <https://redis.io/topics/sentinel>`_.
Meanwhile, the `test` container hosts the code from your checkout of ``redis-py`` and allows running tests against many Python versions.
Docker Tips
^^^^^^^^^^^
Following are a few tips that can help you work with the Docker-based development environment.
To get a bash shell inside of a container:
``$ docker-compose run <service> /bin/bash``
**Note**: The term "service" refers to the "services" defined in the ``docker-compose.yml`` file: "master", "slave", "sentinel_1", "sentinel_2", "sentinel_3", "test".
Containers run a minimal Debian image that probably lacks tools you want to use. To install packages, first get a bash session (see previous tip) and then run:
``$ apt update && apt install <package>``
You can see the combined logging output of all containers like this:
``$ docker-compose logs``
The command `make test` runs all tests in all tested Python environments. To run the tests in a single environment, like Python 3.6, use a command like this:
``$ docker-compose run test tox -e py36 -- --redis-url=redis://master:6379/9``
Here, the flag ``-e py36`` runs tests against the Python 3.6 tox environment. And note from the example that whenever you run tests like this, instead of using `make test`, you need to pass ``-- --redis-url=redis://master:6379/9``. This points the tests at the "master" container.
Our test suite uses ``pytest``. You can run a specific test suite against a specific Python version like this:
``$ docker-compose run test tox -e py36 -- --redis-url=redis://master:6379/9 tests/test_commands.py``
Troubleshooting
^^^^^^^^^^^^^^^
If you get any errors when running ``make dev`` or ``make test``, make sure that you
are using supported versions of Docker and docker-compose.
The included Dockerfiles and docker-compose.yml file work with the following
versions of Docker and docker-compose:
* Docker 19.03.12
* docker-compose 1.26.2
How to Report a Bug
-------------------
Security Vulnerabilities
^^^^^^^^^^^^^^^^^^^^^^^^
**NOTE**: If you find a security vulnerability, do NOT open an issue. Email Andy McCurdy (sedrik@gmail.com) instead.
In order to determine whether you are dealing with a security issue, ask yourself these two questions:
* Can I access something that's not mine, or something I shouldn't have access to?
* Can I disable something for other people?
If the answer to either of those two questions are "yes", then you're probably dealing with a security issue. Note that even if you answer "no" to both questions, you may still be dealing with a security issue, so if you're unsure, just email Andy at sedrik@gmail.com.
Everything Else
^^^^^^^^^^^^^^^
When filing an issue, make sure to answer these five questions:
1. What version of redis-py are you using?
2. What version of redis are you using?
3. What did you do?
4. What did you expect to see?
5. What did you see instead?
How to Suggest a Feature or Enhancement
---------------------------------------
If you'd like to contribute a new feature, make sure you check our issue list to see if someone has already proposed it. Work may already be under way on the feature you want -- or we may have rejected a feature like it already.
If you don't see anything, open a new issue that describes the feature you would like and how it should work.
Code Review Process
-------------------
The core team looks at Pull Requests on a regular basis. We will give feedback as as soon as possible. After feedback, we expect a response within two weeks. After that time, we may close your PR if it isn't showing any activity.
FROM fkrull/multi-python:latest
RUN apt update && apt install -y pypy pypy-dev pypy3-dev
WORKDIR /redis-py
COPY . /redis-py
.PHONY: base clean dev test
base:
docker build -t redis-py-base docker/base
dev: base
docker-compose up -d --build
test: dev
docker-compose run --rm test /redis-py/docker-entry.sh
clean:
docker-compose stop
docker-compose rm -f
......@@ -47,6 +47,12 @@ or from source:
$ python setup.py install
Contributing
------------
Want to contribute a feature, bug report, or report an issue? Check out our `guide to
contributing <https://github.com/andymccurdy/redis-py/blob/master/CONTRIBUTING.rst>`_.
Getting Started
---------------
......
PATH=$PATH:/var/lib/redis/bin
#!/usr/bin/env bash
# need make to build redis
sudo apt-get install make
#!/usr/bin/env bash
source /home/vagrant/redis-py/build_tools/redis_vars.sh
pushd /home/vagrant
uninstall_all_sentinel_instances
uninstall_all_redis_instances
# create a clean directory for redis
rm -rf $REDIS_DIR
mkdir -p $REDIS_BIN_DIR
mkdir -p $REDIS_CONF_DIR
mkdir -p $REDIS_SAVE_DIR
# download, unpack and build redis
mkdir -p $REDIS_DOWNLOAD_DIR
cd $REDIS_DOWNLOAD_DIR
rm -f $REDIS_PACKAGE
rm -rf $REDIS_BUILD_DIR
wget http://download.redis.io/releases/$REDIS_PACKAGE
tar zxvf $REDIS_PACKAGE
cd $REDIS_BUILD_DIR
make
cp src/redis-server $REDIS_DIR/bin
cp src/redis-cli $REDIS_DIR/bin
cp src/redis-sentinel $REDIS_DIR/bin
popd
#!/usr/bin/env bash
source /home/vagrant/redis-py/build_tools/redis_vars.sh
for filename in `ls $VAGRANT_REDIS_CONF_DIR`; do
# cuts the order prefix off of the filename, e.g. 001-master -> master
PROCESS_NAME=redis-`echo $filename | cut -f 2- -d -`
echo "======================================"
echo "INSTALLING REDIS SERVER: $PROCESS_NAME"
echo "======================================"
# make sure the instance is uninstalled (it should be already)
uninstall_instance $PROCESS_NAME
# base config
mkdir -p $REDIS_CONF_DIR
cp $REDIS_BUILD_DIR/redis.conf $REDIS_CONF_DIR/$PROCESS_NAME.conf
# override config values from file
cat $VAGRANT_REDIS_CONF_DIR/$filename >> $REDIS_CONF_DIR/$PROCESS_NAME.conf
# replace placeholder variables in init.d script
cp $VAGRANT_DIR/redis_init_script /etc/init.d/$PROCESS_NAME
sed -i "s/{{ PROCESS_NAME }}/$PROCESS_NAME/g" /etc/init.d/$PROCESS_NAME
# need to read the config file to find out what port this instance will run on
port=`grep port $VAGRANT_REDIS_CONF_DIR/$filename | cut -f 2 -d " "`
sed -i "s/{{ PORT }}/$port/g" /etc/init.d/$PROCESS_NAME
chmod 755 /etc/init.d/$PROCESS_NAME
# and tell update-rc.d about it
update-rc.d $PROCESS_NAME defaults 98
# save the $PROCESS_NAME into installed instances file
echo $PROCESS_NAME >> $REDIS_INSTALLED_INSTANCES_FILE
# start redis
/etc/init.d/$PROCESS_NAME start
done
#!/usr/bin/env bash
source /home/vagrant/redis-py/build_tools/redis_vars.sh
for filename in `ls $VAGRANT_SENTINEL_CONF_DIR`; do
# cuts the order prefix off of the filename, e.g. 001-master -> master
PROCESS_NAME=sentinel-`echo $filename | cut -f 2- -d -`
echo "========================================="
echo "INSTALLING SENTINEL SERVER: $PROCESS_NAME"
echo "========================================="
# make sure the instance is uninstalled (it should be already)
uninstall_instance $PROCESS_NAME
# base config
mkdir -p $REDIS_CONF_DIR
cp $REDIS_BUILD_DIR/sentinel.conf $REDIS_CONF_DIR/$PROCESS_NAME.conf
# override config values from file
cat $VAGRANT_SENTINEL_CONF_DIR/$filename >> $REDIS_CONF_DIR/$PROCESS_NAME.conf
# replace placeholder variables in init.d script
cp $VAGRANT_DIR/sentinel_init_script /etc/init.d/$PROCESS_NAME
sed -i "s/{{ PROCESS_NAME }}/$PROCESS_NAME/g" /etc/init.d/$PROCESS_NAME
# need to read the config file to find out what port this instance will run on
port=`grep port $VAGRANT_SENTINEL_CONF_DIR/$filename | cut -f 2 -d " "`
sed -i "s/{{ PORT }}/$port/g" /etc/init.d/$PROCESS_NAME
chmod 755 /etc/init.d/$PROCESS_NAME
# and tell update-rc.d about it
update-rc.d $PROCESS_NAME defaults 99
# save the $PROCESS_NAME into installed instances file
echo $PROCESS_NAME >> $SENTINEL_INSTALLED_INSTANCES_FILE
# start redis
/etc/init.d/$PROCESS_NAME start
done
pidfile /var/run/redis-master.pid
bind *
port 6379
daemonize yes
unixsocket /tmp/redis_master.sock
unixsocketperm 777
dbfilename master.rdb
dir /var/lib/redis/backups
pidfile /var/run/redis-slave.pid
bind *
port 6380
daemonize yes
unixsocket /tmp/redis-slave.sock
unixsocketperm 777
dbfilename slave.rdb
dir /var/lib/redis/backups
slaveof 127.0.0.1 6379
#!/bin/sh
### BEGIN INIT INFO
# Provides: redis-server
# Required-Start: $syslog
# Required-Stop: $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start redis-server at boot time
# Description: Control redis-server.
### END INIT INFO
REDISPORT={{ PORT }}
PIDFILE=/var/run/{{ PROCESS_NAME }}.pid
CONF=/var/lib/redis/conf/{{ PROCESS_NAME }}.conf
EXEC=/var/lib/redis/bin/redis-server
CLIEXEC=/var/lib/redis/bin/redis-cli
case "$1" in
start)
if [ -f $PIDFILE ]
then
echo "$PIDFILE exists, process is already running or crashed"
else
echo "Starting Redis server..."
$EXEC $CONF
fi
;;
stop)
if [ ! -f $PIDFILE ]
then
echo "$PIDFILE does not exist, process is not running"
else
PID=$(cat $PIDFILE)
echo "Stopping ..."
$CLIEXEC -p $REDISPORT shutdown
while [ -x /proc/${PID} ]
do
echo "Waiting for Redis to shutdown ..."
sleep 1
done
echo "Redis stopped"
fi
;;
*)
echo "Please use start or stop as first argument"
;;
esac
#!/usr/bin/env bash
VAGRANT_DIR=/home/vagrant/redis-py/build_tools
VAGRANT_REDIS_CONF_DIR=$VAGRANT_DIR/redis-configs
VAGRANT_SENTINEL_CONF_DIR=$VAGRANT_DIR/sentinel-configs
REDIS_VERSION=3.2.0
REDIS_DOWNLOAD_DIR=/home/vagrant/redis-downloads
REDIS_PACKAGE=redis-$REDIS_VERSION.tar.gz
REDIS_BUILD_DIR=$REDIS_DOWNLOAD_DIR/redis-$REDIS_VERSION
REDIS_DIR=/var/lib/redis
REDIS_BIN_DIR=$REDIS_DIR/bin
REDIS_CONF_DIR=$REDIS_DIR/conf
REDIS_SAVE_DIR=$REDIS_DIR/backups
REDIS_INSTALLED_INSTANCES_FILE=$REDIS_DIR/redis-instances
SENTINEL_INSTALLED_INSTANCES_FILE=$REDIS_DIR/sentinel-instances
function uninstall_instance() {
# Expects $1 to be the init.d filename, e.g. redis-nodename or
# sentinel-nodename
if [ -a /etc/init.d/$1 ]; then
echo "======================================"
echo "UNINSTALLING REDIS SERVER: $1"
echo "======================================"
/etc/init.d/$1 stop
update-rc.d -f $1 remove
rm -f /etc/init.d/$1
fi;
rm -f $REDIS_CONF_DIR/$1.conf
}
function uninstall_all_redis_instances() {
if [ -a $REDIS_INSTALLED_INSTANCES_FILE ]; then
cat $REDIS_INSTALLED_INSTANCES_FILE | while read line; do
uninstall_instance $line;
done;
fi
}
function uninstall_all_sentinel_instances() {
if [ -a $SENTINEL_INSTALLED_INSTANCES_FILE ]; then
cat $SENTINEL_INSTALLED_INSTANCES_FILE | while read line; do
uninstall_instance $line;
done;
fi
}
pidfile /var/run/sentinel-1.pid
port 26379
daemonize yes
# short timeout for sentinel tests
sentinel down-after-milliseconds mymaster 500
pidfile /var/run/sentinel-2.pid
port 26380
daemonize yes
# short timeout for sentinel tests
sentinel down-after-milliseconds mymaster 500
pidfile /var/run/sentinel-3.pid
port 26381
daemonize yes
# short timeout for sentinel tests
sentinel down-after-milliseconds mymaster 500
#!/bin/sh
### BEGIN INIT INFO
# Provides: redis-sentintel
# Required-Start: $syslog
# Required-Stop: $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start redis-sentinel at boot time
# Description: Control redis-sentinel.
### END INIT INFO
SENTINELPORT={{ PORT }}
PIDFILE=/var/run/{{ PROCESS_NAME }}.pid
CONF=/var/lib/redis/conf/{{ PROCESS_NAME }}.conf
EXEC=/var/lib/redis/bin/redis-sentinel
CLIEXEC=/var/lib/redis/bin/redis-cli
case "$1" in
start)
if [ -f $PIDFILE ]
then
echo "$PIDFILE exists, process is already running or crashed"
else
echo "Starting Redis Sentinel..."
$EXEC $CONF
fi
;;
stop)
if [ ! -f $PIDFILE ]
then
echo "$PIDFILE does not exist, process is not running"
else
PID=$(cat $PIDFILE)
echo "Stopping ..."
$CLIEXEC -p $SENTINELPORT shutdown
while [ -x /proc/${PID} ]
do
echo "Waiting for Sentinel to shutdown ..."
sleep 1
done
echo "Sentinel stopped"
fi
;;
*)
echo "Please use start or stop as first argument"
;;
esac
version: "3.8"
services:
master:
build: docker/master
ports:
- "6379:6379"
slave:
build: docker/slave
depends_on:
- "master"
ports:
- "6380:6380"
sentinel_1:
build: docker/sentinel_1
depends_on:
- "slave"
ports:
- "26379:26379"
sentinel_2:
build: docker/sentinel_2
depends_on:
- "slave"
ports:
- "26380:26380"
sentinel_3:
build: docker/sentinel_3
depends_on:
- "slave"
ports:
- "26381:26381"
test:
build: .
depends_on:
- "sentinel_3"
environment:
REDIS_MASTER_HOST: master
REDIS_MASTER_PORT: 6379
CI:
CI_BUILD_ID:
CI_BUILD_URL:
CI_JOB_ID:
CODECOV_ENV:
CODECOV_SLUG:
CODECOV_TOKEN:
CODECOV_URL:
SHIPPABLE:
TRAVIS:
TRAVIS_BRANCH:
TRAVIS_COMMIT:
TRAVIS_JOB_ID:
TRAVIS_JOB_NUMBER:
TRAVIS_OS_NAME:
TRAVIS_PULL_REQUEST:
TRAVIS_REPO_SLUG:
TRAVIS_TAG:
VCS_BRANCH_NAME:
VCS_COMMIT_ID:
VCS_PULL_REQUEST:
VCS_SLUG:
VCS_TAG:
#!/bin/bash
# This is the entrypoint for "make test". It invokes Tox. If running
# outside the CI environment, it disables uploading the coverage report to codecov
set -eu
REDIS_MASTER="${REDIS_MASTER_HOST}":"${REDIS_MASTER_PORT}"
echo "Testing against Redis Server: ${REDIS_MASTER}"
# skip the "codecov" env if not running on Travis
if [ -z ${TRAVIS-} ]; then
echo "Skipping codecov"
export TOX_SKIP_ENV="codecov"
fi
# use the wait-for-it util to ensure the server is running before invoking Tox
util/wait-for-it.sh ${REDIS_MASTER} -- tox -- --redis-url=redis://"${REDIS_MASTER}"/9
FROM redis:6.0.5-buster
FROM redis-py-base:latest