提交 b8a414ed 编写于 作者: D Dhi Aurrahman 提交者: wu-sheng

Envoy metrics receiver plugin protos (#2304)

* Add envoy metrics service protos

This patch imports protos from envoyproxy/data-plane-api and
prometheus/client_model to enable metrics service service feature in
SkyWalking OAP.
Signed-off-by: NDhi Aurrahman <dio@tetrate.io>

* Add Envoy related docs
Signed-off-by: NDhi Aurrahman <dio@tetrate.io>

* Update the LICENSE to include new protos license

From:
- envoyproxy/data-plane-api
- prometheus/client_model
Signed-off-by: NDhi Aurrahman <dio@tetrate.io>

* Add a complete config.yaml example
Signed-off-by: NDhi Aurrahman <dio@tetrate.io>

* Add note on Envoy dynamic config
Signed-off-by: NDhi Aurrahman <dio@tetrate.io>

* Add protoc-gen-validate license
Signed-off-by: NDhi Aurrahman <dio@tetrate.io>

* Add samples
Signed-off-by: NDhi Aurrahman <dio@tetrate.io>

* Make CI and license right.

* Update README.md

@dhi typo :P

* Update README.md

Another typo :P

* Delete metrics.json

metrics.json is included in identify.json and not be linked in document.
上级 3ed7d705
......@@ -217,6 +217,10 @@ The text of each license is the standard Apache 2.0 license.
proto files from istio/istio: https://github.com/istio/istio Apache 2.0
proto files from istio/api: https://github.com/istio/api Apache 2.0
proto files from envoyproxy/data-plane-api: https://github.com/envoyproxy/data-plane-api Apache 2.0
proto files from prometheus/client_model: https://github.com/prometheus/client_model Apache 2.0
proto files from lyft/protoc-gen-validate: https://github.com/lyft/protoc-gen-validate Apache 2.0
proto files from
========================================================================
......
......@@ -3,7 +3,7 @@ Apache SkyWalking
<img src="http://skywalking.apache.org/assets/logo.svg" alt="Sky Walking logo" height="90px" align="right" />
**SkyWalking**: an APM(application performance monitor) system, especially designed for
**SkyWalking**: an APM(application performance monitor) system, especially designed for
microservices, cloud native and container-based (Docker, Kubernetes, Mesos) architectures.
[![GitHub stars](https://img.shields.io/github/stars/apache/incubator-skywalking.svg?style=for-the-badge&label=Stars&logo=github)](https://github.com/apache/incubator-skywalking)
......@@ -17,7 +17,7 @@ microservices, cloud native and container-based (Docker, Kubernetes, Mesos) arch
# Abstract
**SkyWalking** is an open source APM system, including monitoring, tracing, diagnosing capabilities for distributed system
in Cloud Native architecture.
in Cloud Native architecture.
The core features are following.
- Service, service instance, endpoint metrics analysis
......@@ -34,15 +34,16 @@ The core features are following.
<img src="http://skywalking.apache.org/assets/frame.jpeg"/>
SkyWalking supports to collect telemetry (traces and metrics) data from multiple sources
and multiple formats,
including
and multiple formats,
including
1. Java, .NET Core, NodeJS and PHP auto-instrument agents in SkyWalking format
1. Istio telemetry format
1. Zipkin v1/v2 formats
1. Zipkin v1/v2 format
1. Envoy metrics format (the metric entries itself is prometheus client [metric family](https://github.com/prometheus/client_model/blob/fd36f4220a901265f90734c3183c5f0c91daa0b8/metrics.proto#L77))
# Document
- [6.x Documents](docs/README.md).
- [6.x Documents](docs/README.md).
# Downloads
......@@ -50,7 +51,7 @@ Please head to the [releases page](http://skywalking.apache.org/downloads/) to d
# Code of conduct
This project adheres to the Contributor Covenant [code of conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code.
This project adheres to the Contributor Covenant [code of conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code.
Please report unacceptable behavior to dev@skywalking.apache.org .
# Live Demo
......@@ -103,7 +104,7 @@ SkyWalking enriches the <a href="https://landscape.cncf.io/landscape=observabili
</p>
<p align="center">
<a href="https://openapm.io"><img src="https://openapm.io/static/media/openapm_logo.svg" width="100"/></a>
<a href="https://openapm.io"><img src="https://openapm.io/static/media/openapm_logo.svg" width="100"/></a>
<br/>Our project enriches the <a href="https://openapm.io">OpenAPM Landscape!</a>
</p>
......
......@@ -307,6 +307,9 @@ The text of each license is the standard Apache 2.0 license.
consul-client 1.2.6: https://github.com/rickfast/consul-client, Apache 2.0
okhttp 3.9.0: https://github.com/square/okhttp, Apache 2.0
prometheus client_java 0.6.0: https://github.com/prometheus/client_java, Apache 2.0
proto files from envoyproxy/data-plane-api: https://github.com/envoyproxy/data-plane-api Apache 2.0
proto files from prometheus/client_model: https://github.com/prometheus/client_model Apache 2.0
proto files from lyft/protoc-gen-validate: https://github.com/lyft/protoc-gen-validate Apache 2.0
========================================================================
MIT licenses
......@@ -336,7 +339,7 @@ The text of each license is also included at licenses/LICENSE-[project].txt.
Google: protobuf-java 3.4.0: https://github.com/google/protobuf/blob/master/java/pom.xml , BSD-3-Clause
Google: protobuf-java-util 3.4.0: https://github.com/google/protobuf/blob/master/java/pom.xml , BSD-3-Clause
reflectasm 1.11.3: https://github.com/EsotericSoftware/reflectasm , BSD-3-Clause
gogo-protobuf, proto files: https://github.com/gogo/protobuf BSD-2-Clause
========================================================================
......
......@@ -8,7 +8,7 @@ Setup based on which kind of probes are you going to use. If you don't understan
## Download official releases
- Backend, UI and Java agent are Apache official release, you could find them at [Apache SkyWalking DOWNLOAD page](http://skywalking.apache.org/downloads/).
## Language agents in Service
## Language agents in Service
- [Java agent](service-agent/java-agent/README.md). Introduce how to install java agent to your service, without change any codes.
......@@ -21,7 +21,8 @@ You could go to their project repositories to find out the releases and how to u
## On Service Mesh
- Istio
- [SkyWalking on Istio](istio/README.md). Introduce how to use Istio Mixer bypass Adapter to work with SkyWalking.
- [Envoy Proxy](https://www.envoyproxy.io/)
- [Sending metrics to Skywalking from Envoy](envoy/README.md). How to send metrics from Envoy to SkyWalking using [Metrics service](https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/metrics/v2/metrics_service.proto.html).
## Setup backend
Follow [backend and UI setup document](backend/backend-ui-setup.md) to understand and config the backend for different
......
# Configuring Envoy to send metrics to SkyWalking
In order to let Envoy to send metrics to SkyWalking, we need to feed Envoy with a configuration which contains `stats_sinks` that includes `envoy.metrics_service`.
This `envoy.metrics_service` should be configured as a [`config.grpc_service`](https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/core/grpc_service.proto#envoy-api-msg-core-grpcservice) entry.
The interesting parts of the config is shown in the config below:
```yaml
stats_sinks:
- name: envoy.metrics_service
config:
grpc_service:
# Note: we can use google_grpc implementation as well.
envoy_grpc:
cluster_name: skywalking_service
static_resources:
...
clusters:
- name: skywalking_service
connect_timeout: 5s
type: LOGICAL_DNS
http2_protocol_options: {}
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: service_stats
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: skywalking
# This is the port where SkyWalking serving the Envoy Metrics Service gRPC stream.
port_value: 11800
```
A more complete static configuration, can be observed here [config.yaml].
Note that Envoy can also be configured dynamically through [xDS Protocol](https://github.com/envoyproxy/data-plane-api/blob/master/XDS_PROTOCOL.md).
# Metrics data
Some of the Envoy statitics are listed in this [list](https://www.envoyproxy.io/docs/envoy/latest/configuration/statistics). While a sample data that contains idetifier can be found [here](identify.json), while the metrics only can be observed [here](https://www.envoyproxy.io/docs/envoy/latest/configuration/statistics).
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
protocol: TCP
address: 127.0.0.1
port_value: 9901
stats_sinks:
- name: envoy.metrics_service
config:
grpc_service:
envoy_grpc:
cluster_name: service_skywalking
static_resources:
listeners:
- name: listener_0
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/"
route:
host_rewrite: www.google.com
cluster: service_google
http_filters:
- name: envoy.router
clusters:
- name: service_skywalking
connect_timeout: 5s
type: LOGICAL_DNS
http2_protocol_options: {}
# Comment out the following line to test on v6 networks
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: service_skywalking
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: skywalking
port_value: 11800
- name: service_google
connect_timeout: 5s
type: LOGICAL_DNS
# Comment out the following line to test on v6 networks
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: service_google
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: www.google.com
port_value: 443
tls_context:
sni: www.google.com
{
"identifier": {
"node": {
"buildVersion": "e8ecf08f92941c4a2a892efc9b7dd672cfbb3fe8/1.10.0-dev/Modified/DEBUG/BoringSSL"
}
},
"envoyMetrics": [
{
"name": "cluster.service_google.membership_change",
"metric": [
{
"counter": {
"value": 1
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_google.update_attempt",
"metric": [
{
"counter": {
"value": 60
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_google.update_success",
"metric": [
{
"counter": {
"value": 60
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_rq_pending_total",
"metric": [
{
"counter": {
"value": 59
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.internal.upstream_rq_completed",
"metric": [
{
"counter": {
"value": 59
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.internal.upstream_rq_5xx",
"metric": [
{
"counter": {
"value": 59
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_rq_503",
"metric": [
{
"counter": {
"value": 59
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_rq_5xx",
"metric": [
{
"counter": {
"value": 59
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.update_attempt",
"metric": [
{
"counter": {
"value": 61
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_cx_tx_bytes_total",
"metric": [
{
"counter": {
"value": 2080
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_cx_destroy_remote_with_active_rq",
"metric": [
{
"counter": {
"value": 1
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_cx_total",
"metric": [
{
"counter": {
"value": 59
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.membership_change",
"metric": [
{
"counter": {
"value": 1
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_cx_rx_bytes_total",
"metric": [
{
"counter": {
"value": 48
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_cx_destroy_with_active_rq",
"metric": [
{
"counter": {
"value": 1
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.internal.upstream_rq_503",
"metric": [
{
"counter": {
"value": 59
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_rq_completed",
"metric": [
{
"counter": {
"value": 59
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.update_success",
"metric": [
{
"counter": {
"value": 61
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_cx_http2_total",
"metric": [
{
"counter": {
"value": 59
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_rq_total",
"metric": [
{
"counter": {
"value": 1
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_rq_pending_failure_eject",
"metric": [
{
"counter": {
"value": 59
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_cx_connect_fail",
"metric": [
{
"counter": {
"value": 58
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "http.async-client.rq_total",
"metric": [
{
"counter": {
"value": 59
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "listener_manager.listener_added",
"metric": [
{
"counter": {
"value": 1
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "listener_manager.listener_create_success",
"metric": [
{
"counter": {
"value": 4
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster_manager.cluster_added",
"metric": [
{
"counter": {
"value": 2
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "runtime.deprecated_feature_use",
"metric": [
{
"counter": {
"value": 18
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_google.membership_degraded",
"type": "GAUGE",
"metric": [
{
"gauge": {},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_google.membership_healthy",
"type": "GAUGE",
"metric": [
{
"gauge": {
"value": 1
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_google.membership_total",
"type": "GAUGE",
"metric": [
{
"gauge": {
"value": 1
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.membership_total",
"type": "GAUGE",
"metric": [
{
"gauge": {
"value": 1
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.membership_healthy",
"type": "GAUGE",
"metric": [
{
"gauge": {
"value": 1
},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.membership_degraded",
"type": "GAUGE",
"metric": [
{
"gauge": {},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_rq_active",
"type": "GAUGE",
"metric": [
{
"gauge": {},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_rq_pending_active",
"type": "GAUGE",
"metric": [
{
"gauge": {},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.circuit_breakers.default.rq_open",
"type": "GAUGE",
"metric": [
{
"gauge": {},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.upstream_cx_active",
"type": "GAUGE",
"metric": [
{
"gauge": {},
"timestampMs": "1551412080125"
}
]
},
{
"name": "cluster.service_stats.circuit_breakers.default.rq_pending_open",
"type": "GAUGE",