-[Overview](kubesphere-logging.md#Overview): Explains the architecture and key components behind the logging system.
-[Log Collection and Forward](kubesphere-logging.md#Log-Collection-and-Forward): Introduces logging agent and the sidecar-based solution for collecting application logs on disk.
-[Log Store and Management](kubesphere-logging.md#Log-Store-and-Management): Introduces long-term log storage and management.
-[Log Search, Export and Visualization](kubesphere-logging.md#Log-Search,-Export-and-Visualization): Introduces log query and reporting.
The KubeSphere logging system comprises many components that work together to achieve centralized log management. Once you enable the logging module during installation, a few components will be deployed or created within the namespace `kubesphere-logging-system`:
- Fluent Bit: log collector deployed on each node.
- Fluent-bit Operator: automates tasks for operating Fluent bit instances.
- Logsidecar Injector: injects sidecar into pods for streaming on-disk files to stdout.
- Elasticsearch Cluster: log storage backend.
- Curator: performs stale log cleanup.
The diagram below presents the overall logging architecture:
The following sections will detail the design and functionality of each component.
## Log Collection and Forward
There are two types of logs produced by containers:
- stdout and stderr: written to `.log` files in the host's directory `/var/log/containers` by default.
- application logs: files on the pod's volume, file locations may vary.
Fluent Bit serves as the logging agent. It can access host's container log directory and streams log files to stdout. In KubeSphere, we forked and modified the origin Fluent Bit project fluent/fluent-bit. The difference between fluent/fluent-bit and [kubesphere/fluent-bit](https://github.com/kubesphere/fluent-bit) is that we add a reload interface to support dynamic configuration (see issue [#365](https://github.com/fluent/fluent-bit/issues/365)).
As shown in the diagram below, Fluent-bit controller implements the reload interface and is responsible for the life cycle of fluent-bit process.
The approach to collect container logs on a pod's volume is using a log forwarder sidecar. The sidecar container forwards on-disk application logs to its own stdout. In this way, KubeSphere manages log collection in a unified manner. Besides, [Logsidecar Injector](https://github.com/kubesphere/logsidecar-injector) comes to automate sidecar injection. It takes advantages of `MutatingAdmissionWebhook` to control log forwarder sidecar injection during pod creation.
Though Fluent Bit sends log to Elasticsearch by default in KubeSphere, you may change its output via editing the custom resource defined by the CustomResourceDefinition `fluentbits.logging.kubesphere.io`. The custom resource holds Fluent Bit input/filter/output configurations. Once changes detected, [Fluent-bit operator](https://github.com/kubesphere/fluentbit-operator) will trigger reloading and the new configuration will be used. The Operator pattern is the encouraged way to implement declarative configuration in Kubernetes.
## Log Store and Management
Fluent Bit supports a bunch of [output plugins](https://docs.fluentbit.io/manual/output) that you can choose from to configure how logs will be stored or forwarded. On KubeSphere console, you can easily setup [elasticsearch](https://docs.fluentbit.io/manual/output/elasticsearch), [kafka](https://docs.fluentbit.io/manual/output/kafka) and [fluentd](https://docs.fluentbit.io/manual/output/forward) as outputs.
Take Elasticsearch output for example. The default configuration writes that indices will be created daily with the prefix `ks-logstash-log`. It uses builtin Elasticsearch cluster whose address is `elasticsearch-logging-data.kubesphere-logging-system.svc`. For more details, please explore the custom resource `fluent-bit` (`kubectl get fluentbits.logging.kubesphere.io -n kubesphere-logging-system fluent-bit -oyaml`) .
To clean up stale logs, the Elasticsearch curator cronjob performs a daily check at the cluster level. See [Curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) for more information.
## Log Search, Export and Visualization
KubeSphere wraps Elasticsearch Search and Scroll APIs to support multi-tenancy. Users can use KubeSphere console to search and analyze logs as well as exporting logs to files.
@@ -78,7 +78,9 @@ The function of KubeSphere DevOps is mainly provided by KubeSphere apiserver.
### KubeSphere Logging developer
TODO(@huanggze)
1. Read kubesphere's [Concepts And Designs for Logging](../concepts-and-designs/kubesphere-logging.md). Understand KubeSphere's logging stack.
2. Find the component you are interested in and start from its own repo: [Fluent Bit](https://github.com/kubesphere/fluent-bit), [Fluent-bit Operator](https://github.com/kubesphere/fluentbit-operator) and [Logsidecar Injector](https://github.com/kubesphere/logsidecar-injector).
3. For developers who are interested in KubeSphere logging backend, read [Development Guide for Logging](kubesphere-logging-devlopment-guide.md) and [API doc](https://kubesphere.com.cn/docs/v2.1/api/kubesphere#tag/Log-Query).
This documentation contains backend development guides for interaction with key components behind KubeSphere logging system. Logging backend provides the capabilities of:
- Log search
- Log export
- Log output configuration
- Multi-tenant isolation
## File Tree
The listing below covers all folders related to the logging backend.
```
/pkg
├─api
│ └─logging # declares structs for api responses
│ └─v1alpha2
├─apiserver # implements handler for http requests
│ ├─logging
│ └─tenant
├─kapis # registers APIs and routing
│ ├─logging
│ │ ├─install
│ │ └─v1alpha2
│ ├─tenant
│ │ ├─install
│ │ └─v1alpha2
├─models
│ ├─log # constants, utils and fluent-bit crd operation
│ │ ├─constants.go
│ │ ├─logcollector.go # some utils
│ │ ├─logcrd.go # interacts with fluent-bit crd
│ │ └─types.go
│ └─tenant
└─simple
├─factory.go # contains factory functions for es client options
└─client
├─elasticsearch # wraps es search apis
│ ├─esclient.go # constructs es search body
│ ├─interface.go # general interface methods for es clients
│ ├─options.go # es client options
│ └─versions # client code by es versions
│ ├─v5
│ ├─v6
│ └─v7
└─fluentbit # autogenerated client code for fluent-bit crd
```
## API Design
There are two types of APIs in logging. One for log query, and the other for interacting with the CustomResourceDefinition used by [Fluent-bit Operator](https://github.com/kubesphere/fluentbit-operator). For information about CRD and Fluent-bit Operator, please go to its own repo.
To support multi-tenant isolation, KubeSphere's logging query APIs have the format like below, though the underlying logic is using Elastic Search APIs:
```
GET /namespaces/{namespace}/pods/{pod}/containers/{container}
```
KubeSphere API gateway will decode the URL and conduct authorization. A person who doesn't belong to a namespace will be rejected to make a request.
