diff --git a/CHANGES.md b/CHANGES.md index 357ead70f6581b3c799c1dc146963d1cf2a786cc..6fa8c0d464688c664965f86d3305bf06e7983b0a 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -2,203 +2,23 @@ Changes by Version ================== Release Notes. -5.0.0-RC2 ------------------- - -#### Agent Changes - - Support ActiveMQ 5.x - - Support RuntimeContext used out of TracingContext. - - Support Oracle ojdbc8 Plugin. - - Support ElasticSearch client transport 5.2-5.6 Plugin - - Support using agent.config with given path through system properties. - - Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios. - - Fix HTTPComponent client v4 operation name is empty. - - Fix 2 possible NPEs in Spring plugin. - - Fix a possible span leak in SpringMVC plugin. - - Fix NPE in Spring callback plugin. - -#### Collector Changes - - Add GZip support for Zipkin receiver. - - Add new component IDs for nodejs. - - Fix Zipkin span receiver may miss data in request. - - Optimize codes in heatmap calculation. Reduce unnecessary divide. - - Fix NPE in Alarm content generation. - - Fix the precision lost in `ServiceNameService#startTimeMillis`. - - Fix GC count is 0. - - Fix topology breaks when RPC client uses the async thread call. - -#### UI Changes - - Fix UI port can't be set by startup script in Windows. - - Fix Topology self link error. - - Fix stack color mismatch label color in gc time chart. - -#### Documents - - Add users list. - - Fix several document typo. - - Sync the Chinese documents. - - Add OpenAPM badge. - - Add icon/font documents to NOTICE files. - -5.0.0-beta2 +6.0.0-alpha ------------------ -#### UI -> Collector GraphQL query protocol - - Add order and status in trace query. +SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing +tracing system with analysis and visualization capabilities. We are an **Observability Analysis Platform(OAL)**. -#### Agent Changes - - Add SOFA plugin. - - Add witness class for Kafka plugin. - - Add RuntimeContext in Context. - - Fix RuntimeContext fail in Tomcat plugin. - - Fix incompatible for `getPropertyDescriptors` in Spring core. - - Fix spymemcached plugin bug. - - Fix database URL parser bug. - - Fix `StringIndexOutOfBoundsException` when mysql jdbc url without databaseName。 - - Fix duplicate slash in Spring MVC plugin bug. - - Fix namespace bug. - - Fix NPE in Okhttp plugin when connect failed. - - FIx `MalformedURLException` in httpClientComponent plugin. - - Remove unused dependencies in Dubbo plugin. - - Remove gRPC timeout to avoid out of memory leak. - - Rewrite Async http client plugin. - - [Incubating] Add trace custom ignore optional plugin. +The core and most important features in v6 are +1. Support to collect telemetry data from different sources, such as multiple language agents and service mesh. +1. Extensible stream analysis core. Make SQL and cache analysis available in core level, although not +provided in this release. +1. Provide **Observability Analysis Language(OAL)** to make analysis metric customization available. +1. New GraphQL query protocol. Not binding with UI now. +1. UI topology is better now. +1. New alarm core provided. In alpha, only on service related metric. -#### Collector Changes - - Topology query optimization for more than 100 apps. - - Error rate alarm is not triggered. - - Tolerate unsupported segments. - - Support Integer Array, Long Array, String Array, Double Array in streaming data model. - - Support multiple entry span and multiple service name in one segment durtaion record. - - Use BulkProcessor to control the linear writing of data by multiple threads. - - Determine the log is enabled for the DEBUG level before printing message. - - Add `static` modifier to Logger. - - Add AspNet component. - - Filter inactive service in query. - - Support to query service based on Application. - - Fix `RemoteDataMappingIdNotFoundException` - - Exclude component-libaries.xml file in collector-*.jar, make sure it is in `/conf` only. - - Separate a single TTL in minute to in minute, hour, day, month metric and trace. - - Add order and status in trace query. - - Add folder lock to buffer folder. - - Modify operationName search from `match` to `match_phrase`. - - [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats. - - [Incubating] Support sharding-sphere as storage implementor. - -#### UI Changes - - Support login and access control. - - Add new webapp.yml configuration file. - - Modify webapp startup script. - - Link to trace query from Thermodynamic graph - - Add application selector in service view. - - Add order and status in trace query. - -#### Documents - - Add architecture design doc. - - Reformat deploy document. - - Adjust Tomcat deploy document. - - Remove all Apache licenses files in dist release packages. - - Update user cases. - - Update UI licenses. - - Add incubating sections in doc. -[Issues and Pull requests](https://github.com/apache/incubator-skywalking/milestone/28) - -5.0.0-beta +5.x releases ------------------ - -#### UI -> Collector GraphQL query protocol - - Replace all tps to throughtput/cpm(calls per min) - - Add `getThermodynamic` service - - Update version to beta - -#### Agent Changes - - Support TLS. - - Support namespace. - - Support direct link. - - Support token. - - Add across thread toolkit. - - Add new plugin extend machenism to override agent core implementations. - - Fix an agent start up sequence bug. - - Fix wrong gc count. - - Remove system env override. - - Add Spring AOP aspect patch to avoid aop conflicts. - -#### Collector Changes - - Trace query based on timeline. - - Delete JVM aggregation in second. - - Support TLS. - - Support namespace. - - Support token auth. - - Group and aggregate requests based on reponse time and timeline, support Thermodynamic chart query - - Support component librariy setting through yml file for better extendibility. - - Optimize performance. - - Support short column name in ES or other storage implementor. - - Add a new cache module implementor, based on **Caffeine**. - - Support system property override settings. - - Refactor settings initialization. - - Provide collector instrumentation agent. - - Support .NET core component libraries. - - Fix `divide zero` in query. - - Fix `Data don't remove as expected` in ES implementor. - - Add some checks in collector modulization core. - - Add some test cases. - -#### UI Changes - - New trace query UI. - - New Application UI, merge server tab(removed) into applciation as sub page. - - New Topology UI. - - New response time / throughput TopN list. - - Add Thermodynamic chart in overview page. - - Change all tps to cpm(calls per minutes). - - Fix wrong osName in server view. - - Fix wrong startTime in trace view. - - Fix some icons internet requirements. - -#### Documents - - Add TLS document. - - Add namespace document. - - Add direct link document. - - Add token document. - - Add across thread toolkit document. - - Add a FAQ about, `Agent or collector version upgrade`. - - Sync all English document to Chinese. - -[Issues and Pull requests](https://github.com/apache/incubator-skywalking/milestone/24) - -5.0.0-alpha ------------------- - -#### Agent -> Collector protocol - - Remove C++ keywords - - Move **Ref** into Span from Segment - - Add span type, when register an operation - -#### UI -> Collector GraphQL query protocol - - First version protocol - -#### Agent Changes - - Support gRPC 1.x plugin - - Support kafka 0.11 and 1.x plugin - - Support ServiceComb 0.x plugin - - Support optional plugin mechanism. - - Support Spring 3.x and 4.x bean annotation optional plugin - - Support Apache httpcomponent AsyncClient 4.x plugin - - Provide automatic agent daily tests, and release reports [here](https://github.com/SkywalkingTest/agent-integration-test-report). - - Refactor Postgresql, Oracle, MySQL plugin for compatible. - - Fix jetty client 9 plugin error - - Fix async APIs of okhttp plugin error - - Fix log config didn't work - - Fix a class loader error in okhttp plugin - -#### Collector Changes - - Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month. - - Support new GraphQL query protocol - - Support alarm - - Provide a prototype instrument for collector. - - Support node speculate in cluster and application topology. (Provider Node -> Consumer Node) -> (Provider Node -> MQ Server -> Consumer Node) - -#### UI Changes - - New 5.0.0 UI!!! - - [Issues and Pull requests](https://github.com/apache/incubator-skywalking/milestone/17) +You could find all CHANGES of 5.x at [here](https://github.com/apache/incubator-skywalking/blob/5.x/CHANGES.md) diff --git a/README.md b/README.md index 7cbc346bf94fa3f7ae0024e9a8c6137a8ab4b0df..4c0d3dbfd1f88964c4aa2f9de10ac985faadc4d8 100644 --- a/README.md +++ b/README.md @@ -32,9 +32,12 @@ The core features are following. -SkyWalking supports to collect telemetry (traces and metrics) data from different sources, -in order to provide more options -for different scenarios. +SkyWalking supports to collect telemetry (traces and metrics) data from multiple sources +and multiple formats, +including +1. Java, .NET Core and NodeJS auto-instrument agents in SkyWalking format +1. Istio telemetry format +1. Zipkin v1/v2 formats # Document diff --git a/docs/README.md b/docs/README.md index 4c26627884eb9e37a1d91deaa9607311d84c475d..6eaaaf7a4406c3bab9aa362596b1edf2c34eeecc 100644 --- a/docs/README.md +++ b/docs/README.md @@ -7,7 +7,7 @@ From here you can learn all about **SkyWalking**’s architecture, how to deploy SkyWalking. You can learn from here if you want to understand what is going on under our cool features and visualization. - [Setup](en/setup/README.md). Setup contains guides for installing SkyWalking in different scenarios. As a platform, it provides several ways to provide observability, including monitoring and alarm of course. -- [Contribution Guides](en/guides/README.md). Guides are for PPMC, committer or new contributor. At here, you can know how to contribute from beginning. +- [Contributing Guides](en/guides/README.md). Guides are for PPMC, committer or new contributor. At here, you can know how to contribute from beginning. - [Protocols](en/protocols/README.md). Protocols show the communication ways between agents/probes and backend. Anyone, interested in uplink telemetry data, definitely should read this. - [FAQs](en/FAQ/README.md). Include a manifest, including already known setup problems, secondary developments experiments. When diff --git a/docs/en/concepts-and-designs/query-oap.md b/docs/en/concepts-and-designs/query-oap.md index 46ef873a22f7fc9d6716f6a0e62387f8af696ec3..3e729906a049e232d43a207708d3816608bc76fe 100644 --- a/docs/en/concepts-and-designs/query-oap.md +++ b/docs/en/concepts-and-designs/query-oap.md @@ -1,48 +1,38 @@ # Query in OAP -Query is the core feature of OAP for visualization and other higher system. The query matches the metric type. +Query(s) are provided in GraphQL format. All GraphQL definition files are [here](../../../oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol). -There are two types of query provided. -1. Hard codes query implementor -1. Metric style query of implementor +Here are the explanation of these definitions. -## Hard codes -Hard codes query implementor, is for complex logic query, such as: topology map, dependency map, which -most likely relate to mapping mechanism of the node relationship. +## Common Objects +All objects defined in `common.graphqls` are simple common objects, which could be used in any other +`*.graphqls` definition files. Such as, **Duration**, **Step**, **Scope**. -Even so, hard codes implementors are based on metric style query too, just need extra codes to assemble the -results. +## Metadata +Through Metadata query(s) which defined in `metadata.graphqls`, you could have the meta info of Service, Service Instance and Endpoint, +including name, id, relationship. -## Metric style query -Metric style query is based on the given scope and metric name in oal scripts. +## Metric +Metric query(s) in `metric.graphqls` could be used to fetch data from any variable defined in **OAL** scripts. +You could read value or linear trend of the metric variable by the given duration and id. -Metric style query provided in two ways -- GraphQL way. UI uses this directly, and assembles the pages. -- API way. Most for `Hard codes query implementor` to do extra works. +Also, Thermodynamic heatmap is very different with other single value metric, so it is a special +query op. **Thermodynamic** object, a data matrix, will be returned to represent. -### Grammar -``` -Metric.Scope(SCOPE).Func(METRIC_NAME [, PARAM ...]) -``` +## Aggregation Query +Aggregation query(s) in `aggregation.graphs` right now, are most TopN related query(s). You could +get **TopN** service, service instance and endpoint in different ways. -### Scope -**SCOPE** in (`All`, `Service`, `ServiceInst`, `Endpoint`, `ServiceRelation`, `ServiceInstRelation`, `EndpointRelation`). +## Topology Query +Topology query(s) in `topology.graphqls` provide the consistency query no matter what sources do you +get the topology relation. Also, all entity IDs included in topology will be returned too, for your +convenience to do metric query(s). -### Metric name -Metric name is defined in oal script. Such as **EndpointCalls** is the name defined by `EndpointCalls = from(Endpoint.*).sum()`. +## Trace +At beginning and some scenarios, SkyWalking will be considered as a distributed tracing system. So +of course we will provide trace query. In `trace.graphql` you will find the format, it is nearly the +same format of our trace report/uplink protocol, just in GraphQL version. -### Metric Query Function -Metric Query Functions match the Aggregation Function in most cases, but include some order or filter features. -Try to keep the name as same as the aggregation functions. - -Provided functions -- `top` -- `trend` -- `histogram` -- `sum` - -### Example -For `avg` aggregate func, `top` match it, also with parameter[1] of result size and parameter[2] of order -``` -# for Service_avg = from(Service.latency).avg() -Metric.Scope("Service").topn("Service_avg", 10, "desc") -``` \ No newline at end of file +## Alarm +Alarm query(s) in `alarm.graphql` could be get triggered alarms. Although we believe alarm webhook +in alarm settings(see [alarm setting doc](../setup/backend/backend-alarm.md)) will be more useful +and powerful, still we provide query for SkyWalking UI or simple use scenarios. diff --git a/docs/en/setup/README.md b/docs/en/setup/README.md index edd83fa837eb6a5fdfdec97059fa29b41fb9b7b4..e3f5f3f3fff5018204b563ae6adf95f1082568a1 100644 --- a/docs/en/setup/README.md +++ b/docs/en/setup/README.md @@ -1,6 +1,11 @@ # Setup Setup based on which kind of probes are you going to use. If you don't understand, please read [Concepts and Designs](../concepts-and-designs/README.md) first. +## Download official releases +- Backend, UI and Java agent are Apache official release, you could find them at [Apache SkyWalking DOWNLOAD page](http://skywalking.apache.org/downloads/). +- Download .Net agent by following [Getting started document](https://github.com/OpenSkywalking/skywalking-netcore#getting-started). +- Download Nodejs agent by following [English document](https://github.com/OpenSkywalking/skywalking-nodejs/blob/master/docs/README.md#documents). + ## Language agents in Service - [Java agent](service-agent/java-agent/README.md). Introduce how to install java agent to your service, without change any codes. @@ -9,9 +14,12 @@ Setup based on which kind of probes are you going to use. If you don't understan ## On Service Mesh - Istio - - [SkyWalking on Istio](istio/README.md). Introduce how to use Istio Mixer SkyWalking Adapter to work with SkyWalking. + - [SkyWalking on Istio](istio/README.md). Introduce how to use Istio Mixer bypass Adapter to work with SkyWalking. ## Setup backend Follow [backend and UI setup document](backend/backend-ui-setup.md) to understand and config the backend for different -scenarios, and open advanced features. \ No newline at end of file +scenarios, and open advanced features. + +## Changes log +Backend, UI and Java agent changes are available [here](../../../CHANGES.md). \ No newline at end of file diff --git a/docs/en/setup/backend/backend-alarm.md b/docs/en/setup/backend/backend-alarm.md index 2e5e6cf061bb4cfdd473d06a17caad498a320af3..c4ff6f57c74b8054911590a13056cfb03efa34e4 100644 --- a/docs/en/setup/backend/backend-alarm.md +++ b/docs/en/setup/backend/backend-alarm.md @@ -7,8 +7,10 @@ There are two parts in alarm rule definition. ## Rules Alarm rule is constituted by following keys - **Rule name**. Unique name, show in alarm message. Must end with `_rule`. -- **Indicator name**。A.K.A. metric name in oal script. Only long, double, int types are supported. See -[List of all potential indicator](#list-of-all-potential-indicator-name). +- **Indicator name**. A.K.A. metric name in oal script. Only long, double, int types are supported. See +[List of all potential metric name](#list-of-all-potential-metric-name). +- **Include names**. The following entity names are included in this rule. Such as Service name, +endpoint name. - **Threshold**. The target value. - **OP**. Operator, support `>`, `<`, `=`. Welcome to contribute all OPs. - **Period**. How long should the alarm rule should be checked. This is a time window, which goes with the @@ -34,7 +36,22 @@ rules: count: 3 # How many times of checks, the alarm keeps silence after alarm triggered, default as same as period. silence-period: 10 + + service_percent_rule: + indicator-name: service_percent + # [Optional] Default, match all services in this indicator + include-names: + - service_a + - service_b + threshold: 85 + op: < + period: 10 + count: 4 ``` -## List of all potential indicator name +## List of all potential metric name +The metric names are defined in official [OAL scripts](../../guides/backend-oal-scripts.md), right now +only metric from **Service** scope could be used in Alarm, we will extend in further versions. + +Submit issue or pull request if you want to support any other scope in alarm. diff --git a/docs/en/setup/backend/backend-ip-port.md b/docs/en/setup/backend/backend-ip-port.md index bccb816cb12f690770b80cc8da619f332eef9bf4..657981570ca529495796adc46d78af27e2b77e1a 100644 --- a/docs/en/setup/backend/backend-ip-port.md +++ b/docs/en/setup/backend/backend-ip-port.md @@ -20,8 +20,8 @@ There are two IP/port pair for gRPC and HTTP rest services. ## Notice ### IP binding In case some users are not familiar with IP binding, you should know, after you did that, -the client could only use this IP to access the service. Such as binding `172.09.13.28`, even you are -in this machine, must use this IP rather than `127.0.0.1` or `localhost` to access the service. +the client could only use this IP to access the service. For example, bind `172.09.13.28`, even you are +in this machine, must use `172.09.13.28` rather than `127.0.0.1` or `localhost` to access the service. ### Module provider specified IP and port The IP and port in core are only default provided by core. But some module provider may provide other diff --git a/docs/en/setup/backend/backend-setting-override.md b/docs/en/setup/backend/backend-setting-override.md new file mode 100644 index 0000000000000000000000000000000000000000..7207cc379e101542395b878971c1a4680364a6d1 --- /dev/null +++ b/docs/en/setup/backend/backend-setting-override.md @@ -0,0 +1,24 @@ +# Setting Override +SkyWalking backend supports setting overrides by system properties. +You could override the settings in `application.yml` + +- System properties key rule +**ModuleName**.**ProviderName**.**SettingKey**. + +- Example + + Override `restHost` in this setting segment +```yaml +core: + default: + restHost: 0.0.0.0 + restPort: 12800 + restContextPath: / + gRPCHost: 0.0.0. + gRPCPort: 11800 +``` + +Use command arg +``` +-core.default.restHost=172.0.4.12 +``` \ No newline at end of file diff --git a/docs/en/setup/backend/backend-setup.md b/docs/en/setup/backend/backend-setup.md index bb3d04523b8b78aefae82895c91cb975d33900c8..fa079276956b93071f5ca2bba7de76e0dde5b15b 100644 --- a/docs/en/setup/backend/backend-setup.md +++ b/docs/en/setup/backend/backend-setup.md @@ -52,6 +52,7 @@ read **Set receivers** document in the [link list](#advanced-feature-document-li After understand the setting file structure, you could choose your interesting feature document. We recommend you to read the feature documents in our following order. +1. [Overriding settings](backend-setting-override.md) in application.yml is supported 1. [IP and port setting](backend-ip-port.md). Introduce how IP and port set and be used. 1. [Cluster management](backend-cluster.md). Guide you to set backend server in cluster mode. 1. [Deploy in kubernetes](backend-k8s.md). Guide you to build and use SkyWalking image, and deploy in k8s. diff --git a/docs/en/setup/backend/backend-storage.md b/docs/en/setup/backend/backend-storage.md index 7f60cef7146c2e6acc7555ecd4471033ea6123f9..50b9e3746d82fd09c20ad5094c599a59325ef805 100644 --- a/docs/en/setup/backend/backend-storage.md +++ b/docs/en/setup/backend/backend-storage.md @@ -6,7 +6,7 @@ use is by changing the `application.yml` - [**ElasticSearch 6**](#elasticsearch-6) ## H2 -Active ElasticSearch 6 as storage, set storage provider to **H2**. Default in distribution package. +Active H2 as storage, set storage provider to **H2**. Default in distribution package. Setting fragment example ```yaml diff --git a/docs/en/setup/service-agent/java-agent/README.md b/docs/en/setup/service-agent/java-agent/README.md index 135227780b4a5042c65ed450682ac1d0164f4b8e..6ecbf1c73d1d13d4b75fb8474fa32148619daa01 100644 --- a/docs/en/setup/service-agent/java-agent/README.md +++ b/docs/en/setup/service-agent/java-agent/README.md @@ -1,10 +1,12 @@ # Setup java agent 1. Find `agent` folder in SkyWalking release package -2. Add `-javaagent:/path/to/skywalking-package/agent/skywalking-agent.jar` to JVM argument. And make sure to add it before the `-jar` argument. +2. Set -New agent package looks like this： +3. Add `-javaagent:/path/to/skywalking-package/agent/skywalking-agent.jar` to JVM argument. And make sure to add it before the `-jar` argument. + +The agent release dist is included in Apache [official release](http://skywalking.apache.org/downloads/).MeshReceiverProvider New agent package looks like this. ``` -+-- skywalking-agent ++-- agent +-- activations apm-toolkit-log4j-1.x-activation.jar apm-toolkit-log4j-2.x-activation.jar diff --git a/docs/en/setup/service-agent/java-agent/Setting-override.md b/docs/en/setup/service-agent/java-agent/Setting-override.md index 2970ad9e405b29449fd06bb696f0ddd85e0b7866..ddfc0d9357128f8c4340d73e332894418cf9e45c 100644 --- a/docs/en/setup/service-agent/java-agent/Setting-override.md +++ b/docs/en/setup/service-agent/java-agent/Setting-override.md @@ -1,38 +1,22 @@ # Setting Override -## Supported version -5.0.0-beta + +In default, SkyWalking provide `agent.config` for agent -_Agent setting override supported since 3.2.5_ - -## What is setting override? -In default, SkyWalking provide `agent.config` for client, and `application.yml` for server settings. - -Setting override means end user can override the settings in these config file, by using system properties. - -## Override priority -System.Properties(-D) > Config file +Setting override means end user can override the settings in these config file, through using system properties. -## Override -### Agent -Use `skywalking.` + key in config file as system properties and envs key, to override the value. + +Use `skywalking.` + key in config file as system properties key, to override the value. - Why need this prefix? The agent system properites and env share with target application, this prefix can avoid variable conflict. - -### Collector -Use key in config file as system properties and envs key, to override the value. - -Example: -- Setting in `application.yml` -```yaml -agent_gRPC: - gRPC: - host: localhost - port: 11800 -``` -- Override port to 31200 by system property, add the following line into startup script. +- Override priority + + System.Properties(-D) > Config file + +- Example + + Override `agent.application_code` by this. ``` --Dagent_gRPC.gRPC.port=31200 +-Dskywalking.agent.application_code=31200 ``` \ No newline at end of file diff --git a/oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol b/oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol index 3a83be79a9c23aad6576ed2a4a04b82de6d7a829..f2e54c2cd3f7fdb2cdc975cf791e1bb1d9aab96e 160000 --- a/oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol +++ b/oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol @@ -1 +1 @@ -Subproject commit 3a83be79a9c23aad6576ed2a4a04b82de6d7a829 +Subproject commit f2e54c2cd3f7fdb2cdc975cf791e1bb1d9aab96e