Prepare new doc for v6 (#1695)

* Replace v5 changes by a link to 5.x branch. New v6 changelog.

* Finish query doc and fix a missing field in alarm doc.

* Add link to official oal in alarm doc.

* Fix words.

* Fix some doc

* Update README.md

* Update README.md

* Update README.md

* Update README.md

CHANGES.md

@@ -2,203 +2,23 @@ Changes by Version
 ==================
 Release Notes.
 
-5.0.0-RC2
-------------------
-
-#### Agent Changes
-  - Support ActiveMQ 5.x
-  - Support RuntimeContext used out of TracingContext.
-  - Support Oracle ojdbc8 Plugin.
-  - Support ElasticSearch client transport 5.2-5.6 Plugin
-  - Support using agent.config with given path through system properties.
-  - Add a new way to transmit the Request and Response, to avoid bugs in Hytrix scenarios.
-  - Fix HTTPComponent client v4 operation name is empty.
-  - Fix 2 possible NPEs in Spring plugin.
-  - Fix a possible span leak in SpringMVC plugin.
-  - Fix NPE in Spring callback plugin.
-  
-#### Collector Changes
-  - Add GZip support for Zipkin receiver.
-  - Add new component IDs for nodejs.
-  - Fix Zipkin span receiver may miss data in request.
-  - Optimize codes in heatmap calculation. Reduce unnecessary divide.
-  - Fix NPE in Alarm content generation.
-  - Fix the precision lost in `ServiceNameService#startTimeMillis`.
-  - Fix GC count is 0.
-  - Fix topology breaks when RPC client uses the async thread call.
-  
-#### UI Changes
-  - Fix UI port can't be set by startup script in Windows.
-  - Fix Topology self link error.
-  - Fix stack color mismatch label color in gc time chart.
-  
-#### Documents
-  - Add users list.
-  - Fix several document typo.
-  - Sync the Chinese documents.
-  - Add OpenAPM badge.
-  - Add icon/font documents to NOTICE files.
-
 
-5.0.0-beta2
+6.0.0-alpha
 ------------------
 
-#### UI -> Collector GraphQL query protocol
-  - Add order and status in trace query. 
+SkyWalking 6 is totally new milestone for the project. At this point, we are not just a distributing
+tracing system with analysis and visualization capabilities. We are an **Observability Analysis Platform(OAL)**.
 
-#### Agent Changes
-  - Add SOFA plugin.
-  - Add witness class for Kafka plugin.
-  - Add RuntimeContext in Context.
-  - Fix RuntimeContext fail in Tomcat plugin. 
-  - Fix incompatible for `getPropertyDescriptors` in Spring core. 
-  - Fix spymemcached plugin bug.
-  - Fix database URL parser bug.
-  - Fix `StringIndexOutOfBoundsException` when mysql jdbc url without databaseName。
-  - Fix duplicate slash in Spring MVC plugin bug.
-  - Fix namespace bug.
-  - Fix NPE in Okhttp plugin when connect failed.
-  - FIx `MalformedURLException` in httpClientComponent plugin. 
-  - Remove unused dependencies in Dubbo plugin.
-  - Remove gRPC timeout to avoid out of memory leak.
-  - Rewrite Async http client plugin.
-  - [Incubating] Add trace custom ignore optional plugin. 
+The core and most important features in v6 are
+1. Support to collect telemetry data from different sources, such as multiple language agents and service mesh.
+1. Extensible stream analysis core. Make SQL and cache analysis available in core level, although not
+provided in this release.
+1. Provide **Observability Analysis Language(OAL)** to make analysis metric customization available.
+1. New GraphQL query protocol. Not binding with UI now.
+1. UI topology is better now.
+1. New alarm core provided. In alpha, only on service related metric.
 
-#### Collector Changes
-  - Topology query optimization for more than 100 apps.
-  - Error rate alarm is not triggered.
-  - Tolerate unsupported segments.
-  - Support Integer Array, Long Array, String Array, Double Array in streaming data model.
-  - Support multiple entry span and multiple service name in one segment durtaion record.
-  - Use BulkProcessor to control the linear writing of data by multiple threads.
-  - Determine the log is enabled for the DEBUG level before printing message.
-  - Add `static` modifier to Logger. 
-  - Add AspNet component.
-  - Filter inactive service in query.
-  - Support to query service based on Application.
-  - Fix `RemoteDataMappingIdNotFoundException`
-  - Exclude component-libaries.xml file in collector-*.jar, make sure it is in `/conf` only. 
-  - Separate a single TTL in minute to in minute, hour, day, month metric and trace.
-  - Add order and status in trace query. 
-  - Add folder lock to buffer folder.
-  - Modify operationName search from `match` to `match_phrase`.
-  - [Incubating] Add Zipkin span receiver. Support analysis Zipkin v1/v2 formats.
-  - [Incubating] Support sharding-sphere as storage implementor.
-  
-#### UI Changes
-  - Support login and access control.
-  - Add new webapp.yml configuration file.
-  - Modify webapp startup script.
-  - Link to trace query from Thermodynamic graph
-  - Add application selector in service view.
-  - Add order and status in trace query.
-  
-#### Documents
-  - Add architecture design doc.
-  - Reformat deploy document. 
-  - Adjust Tomcat deploy document.
-  - Remove all Apache licenses files in dist release packages.
-  - Update user cases.
-  - Update UI licenses.
-  - Add incubating sections in doc.
 
-[Issues and Pull requests](https://github.com/apache/incubator-skywalking/milestone/28)
-  
-5.0.0-beta
+5.x releases
 ------------------
- 
-#### UI -> Collector GraphQL query protocol
-  - Replace all tps to throughtput/cpm(calls per min)
-  - Add `getThermodynamic` service
-  - Update version to beta
- 
-#### Agent Changes
-  - Support TLS.
-  - Support namespace.
-  - Support direct link.
-  - Support token.
-  - Add across thread toolkit.
-  - Add new plugin extend machenism to override agent core implementations.
-  - Fix an agent start up sequence bug.
-  - Fix wrong gc count.
-  - Remove system env override.
-  - Add Spring AOP aspect patch to avoid aop conflicts.
- 
-#### Collector Changes
-  - Trace query based on timeline.
-  - Delete JVM aggregation in second.
-  - Support TLS.
-  - Support namespace.
-  - Support token auth.
-  - Group and aggregate requests based on reponse time and timeline, support Thermodynamic chart query
-  - Support component librariy setting through yml file for better extendibility.
-  - Optimize performance.
-  - Support short column name in ES or other storage implementor.
-  - Add a new cache module implementor, based on **Caffeine**.
-  - Support system property override settings.
-  - Refactor settings initialization.
-  - Provide collector instrumentation agent.
-  - Support .NET core component libraries.
-  - Fix `divide zero` in query.
-  - Fix `Data don't remove as expected` in ES implementor.
-  - Add some checks in collector modulization core.
-  - Add some test cases.
- 
-#### UI Changes
-  - New trace query UI.
-  - New Application UI, merge server tab(removed) into applciation as sub page.
-  - New Topology UI.
-  - New response time / throughput TopN list.
-  - Add Thermodynamic chart in overview page.
-  - Change all tps to cpm(calls per minutes).
-  - Fix wrong osName in server view.
-  - Fix wrong startTime in trace view.
-  - Fix some icons internet requirements.
- 
-#### Documents
-   - Add TLS document.
-   - Add namespace document.
-   - Add direct link document.
-   - Add token document.
-   - Add across thread toolkit document.
-   - Add a FAQ about, `Agent or collector version upgrade`.
-   - Sync all English document to Chinese.
- 
-[Issues and Pull requests](https://github.com/apache/incubator-skywalking/milestone/24)
- 
-5.0.0-alpha
-------------------
- 
-#### Agent -> Collector protocol
- - Remove C++ keywords
- - Move **Ref** into Span from Segment
- - Add span type, when register an operation
-
-#### UI -> Collector GraphQL query protocol
- - First version protocol
- 
-#### Agent Changes
- - Support gRPC 1.x plugin
- - Support kafka 0.11 and 1.x plugin
- - Support ServiceComb 0.x plugin
- - Support optional plugin mechanism.
- - Support Spring 3.x and 4.x bean annotation optional plugin
- - Support Apache httpcomponent AsyncClient 4.x plugin 
- - Provide automatic agent daily tests, and release reports [here](https://github.com/SkywalkingTest/agent-integration-test-report).
- - Refactor Postgresql, Oracle, MySQL plugin for compatible.
- - Fix jetty client 9 plugin error
- - Fix async APIs of okhttp plugin error
- - Fix log config didn't work
- - Fix a class loader error in okhttp plugin
- 
-#### Collector Changes
- - Support metrics analysis and aggregation for application, application instance and service in minute, hour, day and month.
- - Support new GraphQL query protocol
- - Support alarm
- - Provide a prototype instrument for collector.
- - Support node speculate in cluster and application topology. (Provider Node -> Consumer Node) -> (Provider Node -> MQ Server -> Consumer Node)
- 
-#### UI Changes
- - New 5.0.0 UI!!!
- 
- [Issues and Pull requests](https://github.com/apache/incubator-skywalking/milestone/17)
+You could find all CHANGES of 5.x at [here](https://github.com/apache/incubator-skywalking/blob/5.x/CHANGES.md)

README.md

@@ -32,9 +32,12 @@ The core features are following.
 
 <img src="https://skywalkingtest.github.io/page-resources/6-alpha-overview.png"/>
 
-SkyWalking supports to collect telemetry (traces and metrics) data from different sources, 
-in order to provide more options
-for different scenarios. 
+SkyWalking supports to collect telemetry (traces and metrics) data from multiple sources
+and multiple formats, 
+including 
+1. Java, .NET Core and NodeJS auto-instrument agents in SkyWalking format
+1. Istio telemetry format
+1. Zipkin v1/v2 formats
 
 
 # Document

docs/README.md

@@ -7,7 +7,7 @@ From here you can learn all about **SkyWalking**’s architecture, how to deploy
 SkyWalking. You can learn from here if you want to understand what is going on under our cool features and visualization.
 - [Setup](en/setup/README.md). Setup contains guides for installing SkyWalking in different scenarios. As a platform, it provides
 several ways to provide observability, including monitoring and alarm of course. 
-- [Contribution Guides](en/guides/README.md). Guides are for PPMC, committer or new contributor. At here, you can know how to contribute from beginning.
+- [Contributing Guides](en/guides/README.md). Guides are for PPMC, committer or new contributor. At here, you can know how to contribute from beginning.
 - [Protocols](en/protocols/README.md). Protocols show the communication ways between agents/probes and backend. Anyone, interested
 in uplink telemetry data, definitely should read this.
 - [FAQs](en/FAQ/README.md). Include a manifest, including already known setup problems, secondary developments experiments. When 

docs/en/concepts-and-designs/query-oap.md

 # Query in OAP
-Query is the core feature of OAP for visualization and other higher system. The query matches the metric type.
+Query(s) are provided in GraphQL format. All GraphQL definition files are [here](../../../oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol).
 
-There are two types of query provided.
-1. Hard codes query implementor
-1. Metric style query of implementor
+Here are the explanation of these definitions.
 
-## Hard codes
-Hard codes query implementor, is for complex logic query, such as: topology map, dependency map, which 
-most likely relate to mapping mechanism of the node relationship.
+## Common Objects
+All objects defined in `common.graphqls` are simple common objects, which could be used in any other 
+`*.graphqls` definition files. Such as, **Duration**, **Step**, **Scope**.
 
-Even so, hard codes implementors are based on metric style query too, just need extra codes to assemble the 
-results.
+## Metadata
+Through Metadata query(s) which defined in `metadata.graphqls`, you could have the meta info of Service, Service Instance and Endpoint, 
+including name, id, relationship. 
 
-## Metric style query
-Metric style query is based on the given scope and metric name in oal scripts.
+## Metric
+Metric query(s) in `metric.graphqls` could be used to fetch data from any variable defined in **OAL** scripts. 
+You could read value or linear trend of the metric variable by the given duration and id.
 
-Metric style query provided in two ways
-- GraphQL way. UI uses this directly, and assembles the pages.
-- API way. Most for `Hard codes query implementor` to do extra works.
+Also, Thermodynamic heatmap is very different with other single value metric, so it is a special
+query op. **Thermodynamic** object, a data matrix, will be returned to represent.
 
-### Grammar
-```
-Metric.Scope(SCOPE).Func(METRIC_NAME [, PARAM ...])
-```
+## Aggregation Query
+Aggregation query(s) in `aggregation.graphs` right now, are most TopN related query(s). You could 
+get **TopN** service, service instance and endpoint in different ways.
 
-### Scope
-**SCOPE** in (`All`, `Service`, `ServiceInst`, `Endpoint`, `ServiceRelation`, `ServiceInstRelation`, `EndpointRelation`).
+## Topology Query
+Topology query(s) in `topology.graphqls` provide the consistency query no matter what sources do you 
+get the topology relation. Also, all entity IDs included in topology will be returned too, for your 
+convenience to do metric query(s).
 
-### Metric name
-Metric name is defined in oal script. Such as **EndpointCalls** is the name defined by `EndpointCalls = from(Endpoint.*).sum()`.
+## Trace
+At beginning and some scenarios, SkyWalking will be considered as a distributed tracing system. So
+of course we will provide trace query. In `trace.graphql` you will find the format, it is nearly the 
+same format of our trace report/uplink protocol, just in GraphQL version. 
 
-### Metric Query Function
-Metric Query Functions match the Aggregation Function in most cases, but include some order or filter features.
-Try to keep the name as same as the aggregation functions.
-
-Provided functions
-- `top`
-- `trend`
-- `histogram`
-- `sum`
-
-### Example
-For `avg` aggregate func, `top` match it, also with parameter[1] of result size and parameter[2] of order
-```
-# for Service_avg = from(Service.latency).avg()
-Metric.Scope("Service").topn("Service_avg", 10, "desc")
-```
\ No newline at end of file
+## Alarm
+Alarm query(s) in `alarm.graphql` could be get triggered alarms. Although we believe alarm webhook
+in alarm settings(see [alarm setting doc](../setup/backend/backend-alarm.md)) will be more useful
+and powerful, still we provide query for SkyWalking UI or simple use scenarios.

docs/en/setup/README.md

 # Setup
 Setup based on which kind of probes are you going to use. If you don't understand, please read [Concepts and Designs](../concepts-and-designs/README.md) first.
 
+## Download official releases
+- Backend, UI and Java agent are Apache official release, you could find them at [Apache SkyWalking DOWNLOAD page](http://skywalking.apache.org/downloads/).
+- Download .Net agent by following [Getting started document](https://github.com/OpenSkywalking/skywalking-netcore#getting-started).
+- Download Nodejs agent by following [English document](https://github.com/OpenSkywalking/skywalking-nodejs/blob/master/docs/README.md#documents). 
+
 ## Language agents in Service 
 
 - [Java agent](service-agent/java-agent/README.md). Introduce how to install java agent to your service, without change any codes.
@@ -9,9 +14,12 @@ Setup based on which kind of probes are you going to use. If you don't understan
 
 ## On Service Mesh
   - Istio
-    - [SkyWalking on Istio](istio/README.md). Introduce how to use Istio Mixer SkyWalking Adapter to work with SkyWalking.
+    - [SkyWalking on Istio](istio/README.md). Introduce how to use Istio Mixer bypass Adapter to work with SkyWalking.
     
 
 ## Setup backend
 Follow [backend and UI setup document](backend/backend-ui-setup.md) to understand and config the backend for different
-scenarios, and open advanced features.
\ No newline at end of file
+scenarios, and open advanced features.
+
+## Changes log
+Backend, UI and Java agent changes are available [here](../../../CHANGES.md).
\ No newline at end of file

docs/en/setup/backend/backend-alarm.md

@@ -7,8 +7,10 @@ There are two parts in alarm rule definition.
 ## Rules
 Alarm rule is constituted by following keys
 - **Rule name**. Unique name, show in alarm message. Must end with `_rule`.
-- **Indicator name**。A.K.A. metric name in oal script. Only long, double, int types are supported. See
-[List of all potential indicator](#list-of-all-potential-indicator-name).
+- **Indicator name**. A.K.A. metric name in oal script. Only long, double, int types are supported. See
+[List of all potential metric name](#list-of-all-potential-metric-name).
+- **Include names**. The following entity names are included in this rule. Such as Service name,
+endpoint name.
 - **Threshold**. The target value.
 - **OP**. Operator, support `>`, `<`, `=`. Welcome to contribute all OPs.
 - **Period**. How long should the alarm rule should be checked. This is a time window, which goes with the
@@ -34,7 +36,22 @@ rules:
     count: 3
     # How many times of checks, the alarm keeps silence after alarm triggered, default as same as period.
     silence-period: 10
+    
+  service_percent_rule:
+    indicator-name: service_percent
+    # [Optional] Default, match all services in this indicator
+    include-names:
+      - service_a
+      - service_b
+    threshold: 85
+    op: <
+    period: 10
+    count: 4
 ```
 
 
-## List of all potential indicator name
+## List of all potential metric name
+The metric names are defined in official [OAL scripts](../../guides/backend-oal-scripts.md), right now 
+only metric from **Service** scope could be used in Alarm, we will extend in further versions. 
+
+Submit issue or pull request if you want to support any other scope in alarm.

docs/en/setup/backend/backend-ip-port.md

@@ -20,8 +20,8 @@ There are two IP/port pair for gRPC and HTTP rest services.
 ## Notice
 ### IP binding
 In case some users are not familiar with IP binding, you should know, after you did that, 
-the client could only use this IP to access the service. Such as binding `172.09.13.28`, even you are
-in this machine, must use this IP rather than `127.0.0.1` or `localhost` to access the service.
+the client could only use this IP to access the service. For example, bind `172.09.13.28`, even you are
+in this machine, must use `172.09.13.28` rather than `127.0.0.1` or `localhost` to access the service.
 
 ### Module provider specified IP and port
 The IP and port in core are only default provided by core. But some module provider may provide other

docs/en/setup/backend/backend-setting-override.md

+# Setting Override
+SkyWalking backend supports setting overrides by system properties. 
+You could override the settings in `application.yml`
+
+- System properties key rule
+**ModuleName**.**ProviderName**.**SettingKey**.
+
+- Example
+
+  Override `restHost` in this setting segment
+```yaml
+core:
+  default:
+    restHost: 0.0.0.0
+    restPort: 12800
+    restContextPath: /
+    gRPCHost: 0.0.0.
+    gRPCPort: 11800
+```
+
+Use command arg
+```
+-core.default.restHost=172.0.4.12
+```
\ No newline at end of file

docs/en/setup/backend/backend-setup.md

@@ -52,6 +52,7 @@ read **Set receivers** document in the [link list](#advanced-feature-document-li
 After understand the setting file structure, you could choose your interesting feature document.
 We recommend you to read the feature documents in our following order.
 
+1. [Overriding settings](backend-setting-override.md) in application.yml is supported
 1. [IP and port setting](backend-ip-port.md). Introduce how IP and port set and be used.
 1. [Cluster management](backend-cluster.md). Guide you to set backend server in cluster mode.
 1. [Deploy in kubernetes](backend-k8s.md). Guide you to build and use SkyWalking image, and deploy in k8s.

docs/en/setup/backend/backend-storage.md

@@ -6,7 +6,7 @@ use is by changing the `application.yml`
 - [**ElasticSearch 6**](#elasticsearch-6)
 
 ## H2
-Active ElasticSearch 6 as storage, set storage provider to **H2**. Default in distribution package.
+Active H2 as storage, set storage provider to **H2**. Default in distribution package.
 
 Setting fragment example
 ```yaml

docs/en/setup/service-agent/java-agent/README.md

 # Setup java agent
 1. Find `agent` folder in SkyWalking release package
-2. Add `-javaagent:/path/to/skywalking-package/agent/skywalking-agent.jar` to JVM argument. And make sure to add it before the `-jar` argument.
+2. Set 
 
-New agent package looks like this：
+3. Add `-javaagent:/path/to/skywalking-package/agent/skywalking-agent.jar` to JVM argument. And make sure to add it before the `-jar` argument.
+
+The agent release dist is included in Apache [official release](http://skywalking.apache.org/downloads/).MeshReceiverProvider New agent package looks like this.
 ```
-+-- skywalking-agent
++-- agent
     +-- activations
          apm-toolkit-log4j-1.x-activation.jar
          apm-toolkit-log4j-2.x-activation.jar

docs/en/setup/service-agent/java-agent/Setting-override.md

 # Setting Override
-## Supported version
-5.0.0-beta + 
+In default, SkyWalking provide `agent.config` for agent 
 
-_Agent setting override supported since 3.2.5_
-
-## What is setting override?
-In default, SkyWalking provide `agent.config` for client, and `application.yml` for server settings. 
-
-Setting override means end user can override the settings in these config file, by using system properties.
-
-## Override priority
-System.Properties(-D) > Config file
+Setting override means end user can override the settings in these config file, through using system properties.
  
-## Override
-### Agent
-Use `skywalking.` + key in config file as system properties and envs key, to override the value.
+
+Use `skywalking.` + key in config file as system properties key, to override the value.
 
 - Why need this prefix?
 
   The agent system properites and env share with target application, this prefix can avoid variable conflict.
-  
-### Collector
-Use key in config file as system properties and envs key, to override the value.
-
-Example:
-- Setting in `application.yml`
-```yaml
-agent_gRPC:
-  gRPC:
-    host: localhost
-    port: 11800
-```
 
-- Override port to 31200 by system property, add the following line into startup script.
+- Override priority
+
+  System.Properties(-D) > Config file  
+
+- Example
+
+  Override `agent.application_code` by this.
 ```
--Dagent_gRPC.gRPC.port=31200
+-Dskywalking.agent.application_code=31200
 ```
\ No newline at end of file

query-protocol @ f2e54c2c

-Subproject commit 3a83be79a9c23aad6576ed2a4a04b82de6d7a829
+Subproject commit f2e54c2cd3f7fdb2cdc975cf791e1bb1d9aab96e