README.md

    (简体中文|English)




    Build Status Release Issues License Slack

    Motivation

    We consider deploying deep learning inference service online to be a user-facing application in the future. The goal of this project: When you have trained a deep neural net with Paddle, you are also capable to deploy the model online easily. A demo of Paddle Serving is as follows:

    Installation

    We highly recommend you to run Paddle Serving in Docker, please visit Run in Docker. See the document for more docker images.

    # Run CPU Docker
    docker pull hub.baidubce.com/paddlepaddle/serving:latest
    docker run -p 9292:9292 --name test -dit hub.baidubce.com/paddlepaddle/serving:latest
    docker exec -it test bash
    # Run GPU Docker
    nvidia-docker pull hub.baidubce.com/paddlepaddle/serving:latest-cuda9.0-cudnn7
    nvidia-docker run -p 9292:9292 --name test -dit hub.baidubce.com/paddlepaddle/serving:latest-cuda9.0-cudnn7
    nvidia-docker exec -it test bash
    pip install paddle-serving-client 
    pip install paddle-serving-server # CPU
    pip install paddle-serving-server-gpu # GPU

    You may need to use a domestic mirror source (in China, you can use the Tsinghua mirror source, add -i https://pypi.tuna.tsinghua.edu.cn/simple to pip command) to speed up the download.

    If you need install modules compiled with develop branch, please download packages from latest packages list and install with pip install command.

    Packages of paddle-serving-server and paddle-serving-server-gpu support Centos 6/7 and Ubuntu 16/18.

    Packages of paddle-serving-client and paddle-serving-app support Linux and Windows, but paddle-serving-client only support python2.7/3.6/3.7.

    Recommended to install paddle >= 1.8.2.

    Pre-built services with Paddle Serving

    Latest release

    Optical Character Recognition
    Object Detection
    Image Segmentation

    Chinese Word Segmentation
    > python -m paddle_serving_app.package --get_model lac
    > tar -xzf lac.tar.gz
    > python lac_web_service.py lac_model/ lac_workdir 9393 &
    > curl -H "Content-Type:application/json" -X POST -d '{"feed":[{"words": "我爱北京天安门"}], "fetch":["word_seg"]}' http://127.0.0.1:9393/lac/prediction
    {"result":[{"word_seg":"我|爱|北京|天安门"}]}

    Image Classification



    > python -m paddle_serving_app.package --get_model resnet_v2_50_imagenet
    > tar -xzf resnet_v2_50_imagenet.tar.gz
    > python resnet50_imagenet_classify.py resnet50_serving_model &
    > curl -H "Content-Type:application/json" -X POST -d '{"feed":[{"image": "https://paddle-serving.bj.bcebos.com/imagenet-example/daisy.jpg"}], "fetch": ["score"]}' http://127.0.0.1:9292/image/prediction
    {"result":{"label":["daisy"],"prob":[0.9341403245925903]}}

    Quick Start Example

    This quick start example is only for users who already have a model to deploy and we prepare a ready-to-deploy model here. If you want to know how to use paddle serving from offline training to online serving, please reference to Train_To_Service

    Boston House Price Prediction model

    wget --no-check-certificate https://paddle-serving.bj.bcebos.com/uci_housing.tar.gz
    tar -xzf uci_housing.tar.gz

    Paddle Serving provides HTTP and RPC based service for users to access

    HTTP service

    Paddle Serving provides a built-in python module called paddle_serving_server.serve that can start a RPC service or a http service with one-line command. If we specify the argument --name uci, it means that we will have a HTTP service with a url of $IP:$PORT/uci/prediction

    python -m paddle_serving_server.serve --model uci_housing_model --thread 10 --port 9292 --name uci
    Argument Type Default Description
    thread int 4 Concurrency of current service
    port int 9292 Exposed port of current service to users
    name str "" Service name, can be used to generate HTTP request url
    model str "" Path of paddle model directory to be served
    mem_optim_off - - Disable memory / graphic memory optimization
    ir_optim - - Enable analysis and optimization of calculation graph
    use_mkl (Only for cpu version) - - Run inference with MKL

    Here, we use curl to send a HTTP POST request to the service we just started. Users can use any python library to send HTTP POST as well, e.g, requests.

    curl -H "Content-Type:application/json" -X POST -d '{"feed":[{"x": [0.0137, -0.1136, 0.2553, -0.0692, 0.0582, -0.0727, -0.1583, -0.0584, 0.6283, 0.4919, 0.1856, 0.0795, -0.0332]}], "fetch":["price"]}' http://127.0.0.1:9292/uci/prediction

    RPC service

    A user can also start a RPC service with paddle_serving_server.serve. RPC service is usually faster than HTTP service, although a user needs to do some coding based on Paddle Serving's python client API. Note that we do not specify --name here.

    python -m paddle_serving_server.serve --model uci_housing_model --thread 10 --port 9292
    # A user can visit rpc service through paddle_serving_client API
    from paddle_serving_client import Client
    
    client = Client()
    client.load_client_config("uci_housing_client/serving_client_conf.prototxt")
    client.connect(["127.0.0.1:9292"])
    data = [0.0137, -0.1136, 0.2553, -0.0692, 0.0582, -0.0727,
            -0.1583, -0.0584, 0.6283, 0.4919, 0.1856, 0.0795, -0.0332]
    fetch_map = client.predict(feed={"x": data}, fetch=["price"])
    print(fetch_map)
    

    Here, client.predict function has two arguments. feed is a python dict with model input variable alias name and values. fetch assigns the prediction variables to be returned from servers. In the example, the name of "x" and "price" are assigned when the servable model is saved during training.

    Some Key Features of Paddle Serving

    • Integrate with Paddle training pipeline seamlessly, most paddle models can be deployed with one line command.
    • Industrial serving features supported, such as models management, online loading, online A/B testing etc.
    • Distributed Key-Value indexing supported which is especially useful for large scale sparse features as model inputs.
    • Highly concurrent and efficient communication between clients and servers supported.
    • Multiple programming languages supported on client side, such as Golang, C++ and python.

    Document

    New to Paddle Serving

    Developers

    About Efficiency

    FAQ

    Design

    Community

    Slack

    To connect with other users and contributors, welcome to join our Slack channel

    Contribution

    If you want to contribute code to Paddle Serving, please reference Contribution Guidelines

    Feedback

    For any feedback or to report a bug, please propose a GitHub Issue.

    License

    Apache 2.0 License

    项目简介

    A flexible, high-performance carrier for machine learning models(『飞桨』服务化部署框架)

    🚀 Github 镜像仓库 🚀

    源项目地址

    https://github.com/PaddlePaddle/Serving

    发行版本 14

    Release v0.9.0

    全部发行版

    贡献者 36

    全部贡献者

    开发语言

    • C++ 51.6 %
    • Python 27.0 %
    • Shell 8.0 %
    • CMake 6.0 %
    • Go 4.4 %