Provide profile exporter tool (#4416)

apm-dist-es7/src/main/assembly/binary-es7.xml

@@ -75,6 +75,12 @@
             <outputDirectory>/agent</outputDirectory>
         </fileSet>
 
+        <!-- Profile exporter tools -->
+        <fileSet>
+            <directory>${project.basedir}/../tools/profile-exporter</directory>
+            <outputDirectory>/tools/profile-exporter</outputDirectory>
+        </fileSet>
+
         <!-- Release docs and licenses -->
         <fileSet>
             <directory>${project.basedir}/../</directory>

apm-dist/src/main/assembly/binary.xml

@@ -75,6 +75,12 @@
             <outputDirectory>/agent</outputDirectory>
         </fileSet>
 
+        <!-- Profile exporter tools -->
+        <fileSet>
+            <directory>${project.basedir}/../tools/profile-exporter</directory>
+            <outputDirectory>/tools/profile-exporter</outputDirectory>
+        </fileSet>
+
         <!-- Release docs and licenses -->
         <fileSet>
             <directory>${project.basedir}/../</directory>

docs/en/FAQ/README.md

@@ -21,3 +21,4 @@ These are known and common FAQs. We welcome you to contribute yours.
 * ["FORBIDDEN/12/index read-only / allow delete (api)" appears in the log](https://discuss.elastic.co/t/forbidden-12-index-read-only-allow-delete-api/110282)
 * [No data shown and backend replies with "Variable 'serviceId' has coerced Null value for NonNull type 'ID!'"](time-and-timezone.md)
 * [**Unexpected endpoint register** warning after 6.6.0](Unexpected-endpoint-register.md)
+* [Use the profile exporter tool if the profile analysis is not right](../guides/backend-profile-export.md)

docs/en/guides/README.md

@@ -146,6 +146,7 @@ miss any newly-added dependency:
 The performance profile is an enhancement feature in the APM system. We are using the thread dump to estimate the method execution time, rather than adding many local spans. In this way, the resource cost would be much less than using distributed tracing to locate slow method. This feature is suitable in the production environment. The following documents are important for developers to understand the key parts of this feature
 - [Profile data report procotol](https://github.com/apache/skywalking-data-collect-protocol/tree/master/profile) is provided like other trace, JVM data through gRPC.
 - [Thread dump merging mechanism](backend-profile.md) introduces the merging mechanism, which helps the end users to understand the profile report.
+- [Exporter tool of profile raw data](backend-profile-export.md) introduces when the visualization doesn't work well through the official UI, how to package the original profile data, which helps the users report the issue.
 
 ## For release
 [Apache Release Guide](How-to-release.md) introduces to the committer team about doing official Apache version release, to avoid 

docs/en/guides/backend-profile-export.md

+# Exporter tool of profile raw data
+When the visualization doesn't work well through the official UI, users could submit the issue to report. This tool helps the users to package the original profile data for helping the community to locate the issue in the user case. NOTICE, this report includes the class name, method name, line number, etc. Before submit this, please make sure this wouldn't become your system vulnerability.
+
+## Export command line Usage
+1. Set the storage in `tools/profile-exporter/application.yml` file by following your use case.
+1. Prepare data
+    - Profile task id: Profile task id
+    - Trace id: Wrong profiled trace id
+    - Export dir: Directory of the data will export
+1. Enter the Skywalking root path
+1. Execute shell command
+    ```bash
+   bash tools/profile-exporter/profile_exporter.sh --taskid={profileTaskId} --traceid={traceId} {exportDir}
+   ```
+1. The file `{traceId}.tar.gz` will be generated after execution shell.
+
+## Exported data content
+1. `basic.yml`: Contains the complete information of the profiled segments in the trace.
+1. `snapshot.data`: All monitored thread snapshot data in the current segment. 
+
+## Report profile issue
+1. Provide exported data generated from this tool.
+1. Provide span operation name, analyze mode(include/exclude children).
+1. Issue description. (If there have the UI screenshots, it's better)

docs/en/guides/backend-profile.md

@@ -47,3 +47,6 @@ The reason for generating multiple top-level trees is that original data can be
     1. Use the same traversal node logic as in the `Combine stack trees` step. Convert to a GraphQL data structure, and put all nodes into a list for subsequent duration calculations.
     2. Calculate each node's duration in parallel. For each node, sort the sequences, if there are two continuous sequences, the duration should add the duration of these two seq's timestamp.
     3. Calculate each node execution in parallel. For each node, the duration of the current node should minus the time consumed by all children.
+
+## Profile data debug
+Please follow the [exporter tool](backend-profile-export.md#export-command-line-usage) to package profile data. Unzip the profile data and using [analyzer main function](../../../oap-server/server-tools/tool-profile-snapshot-bootstrap/src/test/java/org/apache/skywalking/oap/server/tool/profile/exporter/ProfileExportedAnalyze.java) to run it.
\ No newline at end of file

oap-server/pom.xml

@@ -45,6 +45,7 @@
         <module>exporter</module>
         <module>server-configuration</module>
         <module>server-bootstrap</module>
+        <module>server-tools</module>
     </modules>
 
     <properties>

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleConfig.java

@@ -114,7 +114,7 @@ public class CoreModuleConfig extends ModuleConfig {
      */
     private boolean activeExtraModelColumns = false;
 
-    CoreModuleConfig() {
+    public CoreModuleConfig() {
         this.downsampling = new ArrayList<>();
     }
 

oap-server/server-starter-es7/pom.xml

@@ -45,6 +45,13 @@
             <artifactId>storage-elasticsearch7-plugin</artifactId>
             <version>${project.version}</version>
         </dependency>
+
+        <!-- profile exporter -->
+        <dependency>
+            <groupId>org.apache.skywalking</groupId>
+            <artifactId>tool-profile-snapshot-exporter-es7</artifactId>
+            <version>${project.version}</version>
+        </dependency>
     </dependencies>
 
     <build>

oap-server/server-starter/pom.xml

@@ -56,6 +56,13 @@
             <artifactId>zipkin-receiver-plugin</artifactId>
             <version>${project.version}</version>
         </dependency>
+
+        <!-- profile exporter -->
+        <dependency>
+            <groupId>org.apache.skywalking</groupId>
+            <artifactId>tool-profile-snapshot-exporter</artifactId>
+            <version>${project.version}</version>
+        </dependency>
     </dependencies>
 
     <build>

oap-server/server-tools/pom.xml

+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  ~
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>oap-server</artifactId>
+        <groupId>org.apache.skywalking</groupId>
+        <version>7.0.0-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <artifactId>server-tools</artifactId>
+    <packaging>pom</packaging>
+
+    <modules>
+        <module>profile-exporter</module>
+    </modules>
+
+
+</project>
\ No newline at end of file

oap-server/server-tools/profile-exporter/pom.xml

+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  ~
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>server-tools</artifactId>
+        <groupId>org.apache.skywalking</groupId>
+        <version>7.0.0-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <groupId>org.apache.skywalking</groupId>
+    <artifactId>profile-exporter</artifactId>
+    <version>7.0.0-SNAPSHOT</version>
+    <packaging>pom</packaging>
+
+    <modules>
+        <module>tool-profile-snapshot-server-mock</module>
+        <module>tool-profile-snapshot-bootstrap</module>
+        <module>tool-profile-snapshot-exporter</module>
+        <module>tool-profile-snapshot-exporter-es7</module>
+    </modules>
+
+
+</project>
\ No newline at end of file

oap-server/server-tools/profile-exporter/tool-profile-snapshot-bootstrap/pom.xml

+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  ~
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>profile-exporter</artifactId>
+        <groupId>org.apache.skywalking</groupId>
+        <version>7.0.0-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <groupId>org.apache.skywalking</groupId>
+    <artifactId>tool-profile-snapshot-bootstrap</artifactId>
+    <version>7.0.0-SNAPSHOT</version>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.skywalking</groupId>
+            <artifactId>server-bootstrap</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+
+        <!-- core module -->
+        <dependency>
+            <groupId>org.apache.skywalking</groupId>
+            <artifactId>tool-profile-snapshot-server-mock</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+        <!-- core module -->
+
+        <!-- storage module -->
+        <dependency>
+            <groupId>org.apache.skywalking</groupId>
+            <artifactId>storage-jdbc-hikaricp-plugin</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.skywalking</groupId>
+            <artifactId>storage-influxdb-plugin</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+        <!-- storage module -->
+    </dependencies>
+
+</project>
\ No newline at end of file

oap-server/server-tools/profile-exporter/tool-profile-snapshot-bootstrap/src/main/java/org/apache/skywalking/oap/server/tool/profile/exporter/ExporterConfig.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.exporter;
+
+import lombok.Data;
+
+import java.io.File;
+
+@Data
+public class ExporterConfig {
+
+    // profile task id
+    private String taskId;
+
+    // profiled trace id
+    private String traceId;
+
+    // export to file path
+    private String analyzeResultDist;
+
+    /**
+     * parse config from command line
+     */
+    public static ExporterConfig parse(String[] args) {
+        if (args == null || args.length != 3) {
+            throw new IllegalArgumentException("missing config, please recheck");
+        }
+
+        // build config
+        ExporterConfig config = new ExporterConfig();
+        config.setTaskId(args[0]);
+        config.setTraceId(args[1]);
+        config.setAnalyzeResultDist(args[2]);
+
+        return config;
+    }
+
+    /**
+     * initialize config, such as check dist path
+     */
+    public void init() {
+        File dist = new File(analyzeResultDist);
+        if (!dist.exists()) {
+            dist.mkdirs();
+            return;
+        }
+
+        if (dist.isFile()) {
+            throw new IllegalArgumentException(analyzeResultDist + " must be a directory");
+        }
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-bootstrap/src/main/java/org/apache/skywalking/oap/server/tool/profile/exporter/ProfileSnapshotDumper.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.exporter;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.apm.network.language.profile.ThreadSnapshot;
+import org.apache.skywalking.apm.network.language.profile.ThreadStack;
+import org.apache.skywalking.oap.server.core.profile.ProfileThreadSnapshotRecord;
+import org.apache.skywalking.oap.server.core.query.entity.ProfileAnalyzeTimeRange;
+import org.apache.skywalking.oap.server.core.storage.StorageModule;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.BufferedOutputStream;
+import java.io.FileOutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+@Slf4j
+public class ProfileSnapshotDumper {
+
+    public static final int QUERY_PROFILE_SNAPSHOT_RETRY_COUNT = 3;
+    public static final int QUERY_PROFILE_WRITE_PROCESS_LOG = 3;
+
+    /**
+     * dump snapshots to file
+     */
+    public static File dump(ProfiledBasicInfo basicInfo, ModuleManager manager) throws IOException {
+        IProfileThreadSnapshotQueryDAO snapshotQueryDAO = manager.find(StorageModule.NAME).provider().getService(IProfileThreadSnapshotQueryDAO.class);
+        List<ProfiledBasicInfo.SequenceRange> sequenceRanges = basicInfo.buildSequenceRanges();
+        int rangeCount = sequenceRanges.size();
+
+        String segmentId = basicInfo.getSegmentId();
+        File snapshotFile = new File(basicInfo.getConfig().getAnalyzeResultDist() + File.separator + "snapshot.data");
+
+        // reading data and write to file
+        try (OutputStream outputStream = new BufferedOutputStream(new FileOutputStream(snapshotFile))) {
+            for (int i = 0; i < rangeCount; i++) {
+                List<ProfileThreadSnapshotRecord> records = querySnapshot(segmentId, snapshotQueryDAO, sequenceRanges.get(i));
+                for (ProfileThreadSnapshotRecord record : records) {
+                    // transform to proto data and save it
+                    ThreadSnapshot.newBuilder()
+                            .setStack(ThreadStack.parseFrom(record.getStackBinary()))
+                            .setSequence(record.getSequence())
+                            .setTime(record.getDumpTime())
+                            .build()
+                            .writeDelimitedTo(outputStream);
+                }
+
+                // print process log if need
+                if ((i > 0 && i % QUERY_PROFILE_WRITE_PROCESS_LOG == 0) || i == rangeCount - 1) {
+                    log.info("Dump snapshots process:[{}/{}]:{}%", i + 1, rangeCount, (int) ((double) (i + 1) / rangeCount * 100));
+                }
+            }
+        }
+
+        return snapshotFile;
+    }
+
+    /**
+     * query snapshots with retry mechanism
+     */
+    private static List<ProfileThreadSnapshotRecord> querySnapshot(String segmentId, IProfileThreadSnapshotQueryDAO threadSnapshotQueryDAO, ProfiledBasicInfo.SequenceRange sequenceRange) throws IOException {
+        for (int i = 1; i <= QUERY_PROFILE_SNAPSHOT_RETRY_COUNT; i++) {
+            try {
+                return threadSnapshotQueryDAO.queryRecords(segmentId, sequenceRange.getMin(), sequenceRange.getMax());
+            } catch (IOException e) {
+                if (i == QUERY_PROFILE_SNAPSHOT_RETRY_COUNT) {
+                    throw e;
+                }
+            }
+        }
+        return null;
+    }
+
+    /**
+     * load thread snapshots in appointing time range
+     */
+    public static List<ThreadSnapshot> parseFromFileWithTimeRange(File file, List<ProfileAnalyzeTimeRange> timeRanges) throws IOException {
+        try (final FileInputStream fileInputStream = new FileInputStream(file)) {
+            ThreadSnapshot snapshot;
+            final ArrayList<ThreadSnapshot> data = new ArrayList<>();
+            while ((snapshot = ThreadSnapshot.parseDelimitedFrom(fileInputStream)) != null) {
+                ThreadSnapshot finalSnapshot = snapshot;
+                if (timeRanges.stream().filter(t -> finalSnapshot.getTime() >= t.getStart() && finalSnapshot.getTime() <= t.getEnd()).findFirst().isPresent()) {
+                    data.add(snapshot);
+                }
+            }
+            return data;
+        }
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-bootstrap/src/main/java/org/apache/skywalking/oap/server/tool/profile/exporter/ProfileSnapshotExporterBootstrap.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.exporter;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.library.module.ApplicationConfiguration;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.starter.config.ApplicationConfigLoader;
+
+import java.io.File;
+
+@Slf4j
+public class ProfileSnapshotExporterBootstrap {
+    public static void export(String[] args) {
+        ApplicationConfigLoader configLoader = new ApplicationConfigLoader();
+        ModuleManager manager = new ModuleManager();
+        try {
+            // parse config and init
+            ExporterConfig exporterConfig = ExporterConfig.parse(args);
+            exporterConfig.init();
+
+            // init OAP
+            ApplicationConfiguration applicationConfiguration = configLoader.load();
+            manager.init(applicationConfiguration);
+
+            // prepare basic info
+            ProfiledBasicInfo profiledBaseInfo = ProfiledBasicInfo.build(exporterConfig, manager);
+            log.info("Queried profiled basic info, profiled segment start time:{}, duration:{}, total span count:{}, snapshot count:{}",
+                    profiledBaseInfo.getSegmentStartTime(), profiledBaseInfo.getDuration(), profiledBaseInfo.getProfiledSegmentSpans().size(),
+                    profiledBaseInfo.getMaxSequence() - profiledBaseInfo.getMinSequence());
+
+            // write basic info to file
+            File basicInfoFile = profiledBaseInfo.writeFile();
+            log.info("Write segment info to file:{}", basicInfoFile.getAbsolutePath());
+
+            // query and writing snapshot
+            File snapshotFile = ProfileSnapshotDumper.dump(profiledBaseInfo, manager);
+            log.info("Write snapshot to file:{}", snapshotFile);
+
+            // exit program
+            System.exit(0);
+        } catch (Throwable t) {
+            log.error(t.getMessage(), t);
+            System.exit(1);
+        }
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-bootstrap/src/main/java/org/apache/skywalking/oap/server/tool/profile/exporter/ProfiledBasicInfo.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.exporter;
+
+import lombok.Data;
+import lombok.Getter;
+import org.apache.commons.io.FileUtils;
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.query.ProfileTaskQueryService;
+import org.apache.skywalking.oap.server.core.query.TraceQueryService;
+import org.apache.skywalking.oap.server.core.query.entity.BasicTrace;
+import org.apache.skywalking.oap.server.core.query.entity.Span;
+import org.apache.skywalking.oap.server.core.query.entity.Trace;
+import org.apache.skywalking.oap.server.core.storage.StorageModule;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+import org.yaml.snakeyaml.Yaml;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+@Data
+public class ProfiledBasicInfo {
+
+    public static final int SEQUENCE_RANGE_BATCH_SIZE = 500;
+
+    private ExporterConfig config;
+
+    // profiled segment
+    private String segmentId;
+    private long segmentStartTime;
+    private long segmentEndTime;
+    private int duration;
+
+    // spans
+    private List<Span> profiledSegmentSpans;
+
+    // snapshot sequence
+    private int minSequence;
+    private int maxSequence;
+
+    /**
+     * reading data from storage and build data
+     */
+    public static ProfiledBasicInfo build(ExporterConfig config, ModuleManager manager) throws IOException {
+        ProfiledBasicInfo data = new ProfiledBasicInfo();
+        data.setConfig(config);
+
+        ProfileTaskQueryService taskQueryService = manager.find(CoreModule.NAME).provider().getService(ProfileTaskQueryService.class);
+        TraceQueryService traceQueryService = manager.find(CoreModule.NAME).provider().getService(TraceQueryService.class);
+        IProfileThreadSnapshotQueryDAO threadSnapshotQueryDAO = manager.find(StorageModule.NAME).provider().getService(IProfileThreadSnapshotQueryDAO.class);
+
+        // query and found profiled segment
+        List<BasicTrace> taskTraces = taskQueryService.getTaskTraces(config.getTaskId());
+        BasicTrace profiledTrace = taskTraces.stream().filter(t -> t.getTraceIds().contains(config.getTraceId())).findFirst().orElse(null);
+        if (profiledTrace == null) {
+            throw new IllegalArgumentException("Cannot fount profiled segment in current task: " + config.getTaskId()
+                    + ", segment id: " + config.getTraceId() + ", current task total profiled trace count is " + taskTraces.size());
+        }
+
+        // setting segment basic info
+        String segmentId = profiledTrace.getSegmentId();
+        long startTime = Long.parseLong(profiledTrace.getStart());
+        long endTime = startTime + profiledTrace.getDuration();
+        data.setSegmentId(segmentId);
+        data.setSegmentStartTime(startTime);
+        data.setSegmentEndTime(endTime);
+        data.setDuration(profiledTrace.getDuration());
+
+        // query spans
+        Trace trace = traceQueryService.queryTrace(config.getTraceId());
+        List<Span> profiledSegmentSpans = trace.getSpans().stream().filter(s -> Objects.equals(s.getSegmentId(), segmentId)).collect(Collectors.toList());
+        if (CollectionUtils.isEmpty(profiledSegmentSpans)) {
+            throw new IllegalArgumentException("Current segment cannot found any span");
+        }
+        data.setProfiledSegmentSpans(profiledSegmentSpans);
+
+        // query snapshots sequences
+        int minSequence = threadSnapshotQueryDAO.queryMinSequence(segmentId, startTime, endTime);
+        int maxSequence = threadSnapshotQueryDAO.queryMaxSequence(segmentId, startTime, endTime);
+        data.setMinSequence(minSequence);
+        data.setMaxSequence(maxSequence);
+
+        return data;
+    }
+
+    /**
+     * serialize data to file
+     */
+    public File writeFile() throws IOException {
+        String serialData = new Yaml().dump(this);
+        File file = new File(config.getAnalyzeResultDist() + File.separator + "basic.yml");
+        FileUtils.write(file, serialData, "UTF-8");
+        return file;
+    }
+
+    /**
+     * deserialize data from file
+     */
+    public static ProfiledBasicInfo parseFormFile(File file) throws IOException {
+        try (FileInputStream fileInputStream = new FileInputStream(file)) {
+            return new Yaml().loadAs(fileInputStream, ProfiledBasicInfo.class);
+        }
+    }
+
+    /**
+     * build current profiles segment snapshot search sequence ranges
+     */
+    public List<SequenceRange> buildSequenceRanges() {
+        ArrayList<SequenceRange> ranges = new ArrayList<>();
+        do {
+            int batchMax = Math.min(minSequence + SEQUENCE_RANGE_BATCH_SIZE, maxSequence);
+            ranges.add(new SequenceRange(minSequence, batchMax));
+            minSequence = batchMax;
+        }
+        while (minSequence < maxSequence);
+
+        return ranges;
+    }
+
+    @Getter
+    public static class SequenceRange {
+        private int min;
+        private int max;
+
+        public SequenceRange(int min, int max) {
+            this.min = min;
+            this.max = max;
+        }
+    }
+
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-bootstrap/src/test/java/org/apache/skywalking/oap/server/tool/profile/exporter/ProfileAnalyzeSnapshotDAO.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.exporter;
+
+import com.google.common.primitives.Ints;
+import org.apache.skywalking.apm.network.language.profile.ThreadSnapshot;
+import org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentRecord;
+import org.apache.skywalking.oap.server.core.profile.ProfileThreadSnapshotRecord;
+import org.apache.skywalking.oap.server.core.query.entity.BasicTrace;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileThreadSnapshotQueryDAO;
+
+import java.io.IOException;
+import java.util.Comparator;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class ProfileAnalyzeSnapshotDAO implements IProfileThreadSnapshotQueryDAO {
+
+    private final List<ThreadSnapshot> snapshots;
+
+    public ProfileAnalyzeSnapshotDAO(List<ThreadSnapshot> snapshots) {
+        this.snapshots = snapshots;
+    }
+
+    @Override
+    public List<BasicTrace> queryProfiledSegments(String taskId) throws IOException {
+        return null;
+    }
+
+    @Override
+    public int queryMinSequence(String segmentId, long start, long end) throws IOException {
+        return snapshots.stream().sorted(Comparator.comparingInt(ThreadSnapshot::getSequence)).findFirst().get().getSequence();
+    }
+
+    @Override
+    public int queryMaxSequence(String segmentId, long start, long end) throws IOException {
+        return snapshots.stream().sorted((s1, s2) -> -Ints.compare(s1.getSequence(), s2.getSequence())).findFirst().get().getSequence();
+    }
+
+    @Override
+    public List<ProfileThreadSnapshotRecord> queryRecords(String segmentId, int minSequence, int maxSequence) throws IOException {
+        return snapshots.parallelStream()
+                .filter(s -> s.getSequence() >= minSequence && s.getSequence() < maxSequence)
+                .map(this::buildFromSnapshot)
+                .collect(Collectors.toList());
+    }
+
+    @Override
+    public SegmentRecord getProfiledSegment(String segmentId) throws IOException {
+        return null;
+    }
+
+    private ProfileThreadSnapshotRecord buildFromSnapshot(ThreadSnapshot snapshot) {
+        final ProfileThreadSnapshotRecord record = new ProfileThreadSnapshotRecord();
+        record.setStackBinary(snapshot.getStack().toByteArray());
+        record.setDumpTime(snapshot.getTime());
+        record.setSequence(snapshot.getSequence());
+        return record;
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-bootstrap/src/test/java/org/apache/skywalking/oap/server/tool/profile/exporter/ProfileExportedAnalyze.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.exporter;
+
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.apm.network.language.profile.ThreadSnapshot;
+import org.apache.skywalking.oap.server.core.CoreModuleConfig;
+import org.apache.skywalking.oap.server.core.profile.analyze.ProfileAnalyzer;
+import org.apache.skywalking.oap.server.core.query.entity.ProfileAnalyzation;
+import org.apache.skywalking.oap.server.core.query.entity.ProfileAnalyzeTimeRange;
+import org.apache.skywalking.oap.server.core.query.entity.ProfileStackElement;
+import org.apache.skywalking.oap.server.core.query.entity.ProfileStackTree;
+import org.apache.skywalking.oap.server.core.query.entity.Span;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.library.util.CollectionUtils;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+@Slf4j
+public class ProfileExportedAnalyze {
+
+    public static void main(String[] args) throws IOException {
+        // input
+        final File basicInfoFile = new File("basic.yml");
+        final File snapshotFile = new File("snapshot.data");
+        String profiledSpanName = "";
+        boolean includeChildren = true;
+
+        // parsing data
+        final ProfiledBasicInfo basicInfo = ProfiledBasicInfo.parseFormFile(basicInfoFile);
+        final List<Span> sameNameSpans = basicInfo.getProfiledSegmentSpans().stream().filter(s -> Objects.equals(s.getEndpointName(), profiledSpanName)).collect(Collectors.toList());
+        if (CollectionUtils.isEmpty(sameNameSpans)) {
+            log.warn("Cannot found same name span:{}", profiledSpanName);
+            return;
+        }
+        final Span span = sameNameSpans.get(0);
+
+        // build time ranges
+        final List<ProfileAnalyzeTimeRange> timeRanges = buildTimeRanges(basicInfo, span, includeChildren);
+        final List<ThreadSnapshot> snapshots = ProfileSnapshotDumper.parseFromFileWithTimeRange(snapshotFile, timeRanges);
+        log.info("Total found snapshot count: {}", snapshots.size());
+
+        // analyze and print
+        final ProfileAnalyzer profileAnalyzer = new Analyzer(snapshots);
+        final ProfileAnalyzation analyzation = profileAnalyzer.analyze(null, timeRanges);
+        printAnalyzation(analyzation);
+
+    }
+
+    private static void printAnalyzation(ProfileAnalyzation analyzation) {
+        for (int i = 0; i < analyzation.getTrees().size(); i++) {
+            log.info("--------------------");
+            log.info("tree: {}", i);
+            log.info("--------------------");
+
+            final ProfileStackTree tree = analyzation.getTrees().get(i);
+            final List<ProfileStackElement> elements = tree.getElements();
+
+            printElements(elements, 0, 0);
+        }
+    }
+
+    private static void printElements(List<ProfileStackElement> elements, int depth, int parentId) {
+        final StringBuilder depthPrefix = new StringBuilder();
+        for (int i = 0; i < depth; i++) {
+            depthPrefix.append(" |");
+        }
+        depthPrefix.append("\\-");
+
+        for (ProfileStackElement element : elements) {
+            if (element.getParentId() != parentId) {
+                continue;
+            }
+
+            log.info("{} {}: [count:{}], [duration:{}:{}]", depthPrefix, element.getCodeSignature(), element.getCount(), element.getDuration(), element.getDurationChildExcluded());
+            printElements(elements, depth + 1, element.getId());
+        }
+    }
+
+    private static List<ProfileAnalyzeTimeRange> buildTimeRanges(ProfiledBasicInfo basicInfo, Span currentSpan, boolean includeChildren) {
+        if (includeChildren) {
+            final ProfileAnalyzeTimeRange range = new ProfileAnalyzeTimeRange();
+            range.setStart(currentSpan.getStartTime());
+            range.setEnd(currentSpan.getEndTime());
+            return Collections.singletonList(range);
+        }
+
+        // find children spans
+        final List<Span> childrenSpans = basicInfo.getProfiledSegmentSpans().stream()
+                .filter(s -> s.getParentSpanId() == currentSpan.getSpanId())
+                .sorted(Comparator.comparingLong(Span::getStartTime))
+                .collect(Collectors.toList());
+
+        final ArrayList<ProfileAnalyzeTimeRange> ranges = new ArrayList<>();
+        long startTime = currentSpan.getStartTime();
+        long endTime = currentSpan.getStartTime();
+        for (Span span : childrenSpans) {
+            if (span.getStartTime() > startTime) {
+                final ProfileAnalyzeTimeRange range = new ProfileAnalyzeTimeRange();
+                range.setStart(startTime);
+                range.setEnd(span.getStartTime() - 1);
+                ranges.add(range);
+            }
+
+            startTime = span.getEndTime();
+            endTime = span.getEndTime();
+        }
+
+        // add last range
+        if (endTime != currentSpan.getEndTime()) {
+            final ProfileAnalyzeTimeRange range = new ProfileAnalyzeTimeRange();
+            range.setStart(endTime);
+            range.setEnd(currentSpan.getEndTime());
+            ranges.add(range);
+        }
+
+        return ranges;
+    }
+
+    private static class Analyzer extends ProfileAnalyzer {
+
+        private final IProfileThreadSnapshotQueryDAO dao;
+
+        public Analyzer(List<ThreadSnapshot> snapshots) {
+            super(null, new CoreModuleConfig().getMaxPageSizeOfQueryProfileSnapshot(), new CoreModuleConfig().getMaxSizeOfAnalyzeProfileSnapshot());
+            this.dao = new ProfileAnalyzeSnapshotDAO(snapshots);
+        }
+
+        @Override
+        protected IProfileThreadSnapshotQueryDAO getProfileThreadSnapshotQueryDAO() {
+            return dao;
+        }
+    }
+
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-bootstrap/src/test/java/org/apache/skywalking/oap/server/tool/profile/exporter/test/ExportedData.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.exporter.test;
+
+import lombok.Data;
+
+import java.util.List;
+
+@Data
+public class ExportedData {
+
+    private String taskId;
+    private String segmentId;
+    private int limit;
+    private List<String> snapshots;
+    private List<Span> spans;
+    private String traceId;
+
+    @Data
+    public static class Span {
+        private String operation;
+        private int start;
+        private int end;
+        private List<Span> children;
+        private int id;
+        private int parentId;
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-bootstrap/src/test/java/org/apache/skywalking/oap/server/tool/profile/exporter/test/ProfileExportSnapshotDAO.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.exporter.test;
+
+import org.apache.skywalking.apm.network.language.profile.ThreadStack;
+import org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentRecord;
+import org.apache.skywalking.oap.server.core.profile.ProfileThreadSnapshotRecord;
+import org.apache.skywalking.oap.server.core.query.entity.BasicTrace;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileThreadSnapshotQueryDAO;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+public class ProfileExportSnapshotDAO implements IProfileThreadSnapshotQueryDAO {
+
+    private final ExportedData exportedData;
+
+    public ProfileExportSnapshotDAO(ExportedData exportedData) {
+        this.exportedData = exportedData;
+    }
+
+    @Override
+    public List<BasicTrace> queryProfiledSegments(String taskId) throws IOException {
+        final BasicTrace basicTrace = new BasicTrace();
+        basicTrace.setSegmentId(exportedData.getSegmentId());
+        basicTrace.getTraceIds().add(exportedData.getTraceId());
+        basicTrace.setStart(exportedData.getSpans().get(0).getStart() + "");
+        basicTrace.setDuration(exportedData.getSpans().get(0).getEnd());
+        return Collections.singletonList(basicTrace);
+    }
+
+    @Override
+    public int queryMinSequence(String segmentId, long start, long end) throws IOException {
+        for (int i = 0; i < exportedData.getSnapshots().size(); i++) {
+            if (i * exportedData.getLimit() >= start) {
+                return i;
+            }
+        }
+        return 0;
+    }
+
+    @Override
+    public int queryMaxSequence(String segmentId, long start, long end) throws IOException {
+        for (int i = exportedData.getSnapshots().size() - 1; i >= 0; i--) {
+            if (i * exportedData.getLimit() <= end) {
+                return i + 1;
+            }
+        }
+        return 0;
+    }
+
+    @Override
+    public List<ProfileThreadSnapshotRecord> queryRecords(String segmentId, int minSequence, int maxSequence) throws IOException {
+        final ArrayList<ProfileThreadSnapshotRecord> records = new ArrayList<>();
+        for (int i = 0; i < exportedData.getSnapshots().size(); i++) {
+            if (i >= minSequence && i < maxSequence) {
+                final ProfileThreadSnapshotRecord record = new ProfileThreadSnapshotRecord();
+                record.setSequence(i);
+                record.setDumpTime(i * exportedData.getLimit());
+                final ThreadStack.Builder stack = ThreadStack.newBuilder().addAllCodeSignatures(Arrays.asList(exportedData.getSnapshots().get(i).split("-")));
+                record.setStackBinary(stack.build().toByteArray());
+
+                records.add(record);
+            }
+        }
+        return records;
+    }
+
+    @Override
+    public SegmentRecord getProfiledSegment(String segmentId) throws IOException {
+        return null;
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-bootstrap/src/test/java/org/apache/skywalking/oap/server/tool/profile/exporter/test/ProfileSnapshotExporterTest.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.exporter.test;
+
+import org.apache.skywalking.apm.network.language.profile.ThreadSnapshot;
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.CoreModuleConfig;
+import org.apache.skywalking.oap.server.core.CoreModuleProvider;
+import org.apache.skywalking.oap.server.core.cache.ServiceInventoryCache;
+import org.apache.skywalking.oap.server.core.query.ProfileTaskQueryService;
+import org.apache.skywalking.oap.server.core.query.TraceQueryService;
+import org.apache.skywalking.oap.server.core.query.entity.ProfileAnalyzeTimeRange;
+import org.apache.skywalking.oap.server.core.storage.StorageModule;
+import org.apache.skywalking.oap.server.core.storage.profile.IProfileThreadSnapshotQueryDAO;
+import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
+import org.apache.skywalking.oap.server.library.module.ModuleManager;
+import org.apache.skywalking.oap.server.library.util.ResourceUtils;
+import org.apache.skywalking.oap.server.tool.profile.exporter.ExporterConfig;
+import org.apache.skywalking.oap.server.tool.profile.exporter.ProfileSnapshotDumper;
+import org.apache.skywalking.oap.server.tool.profile.exporter.ProfiledBasicInfo;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.internal.util.reflection.Whitebox;
+import org.powermock.modules.junit4.PowerMockRunner;
+import org.yaml.snakeyaml.Yaml;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.Reader;
+import java.util.Collections;
+import java.util.List;
+
+@RunWith(PowerMockRunner.class)
+public class ProfileSnapshotExporterTest {
+
+    @Mock
+    private CoreModuleProvider moduleProvider;
+    @Mock
+    private ModuleManager moduleManager;
+    @Mock
+    private CoreModuleConfig coreModuleConfig;
+
+    private ExportedData exportedData;
+
+    @Before
+    public void init() throws IOException {
+        CoreModule coreModule = Mockito.spy(CoreModule.class);
+        StorageModule storageModule = Mockito.spy(StorageModule.class);
+        Whitebox.setInternalState(coreModule, "loadedProvider", moduleProvider);
+        Whitebox.setInternalState(storageModule, "loadedProvider", moduleProvider);
+        Mockito.when(moduleManager.find(CoreModule.NAME)).thenReturn(coreModule);
+        Mockito.when(moduleManager.find(StorageModule.NAME)).thenReturn(storageModule);
+        final ProfileTaskQueryService taskQueryService = new ProfileTaskQueryService(moduleManager, coreModuleConfig);
+
+        Mockito.when(moduleProvider.getService(ProfileTaskQueryService.class)).thenReturn(taskQueryService);
+        Mockito.when(moduleProvider.getService(TraceQueryService.class)).thenReturn(new TraceQueryService(moduleManager));
+
+        try (final Reader reader = ResourceUtils.read("profile.yml");) {
+            exportedData = new Yaml().loadAs(reader, ExportedData.class);
+        }
+        final ServiceInventoryCache serviceInventoryCache = new ServiceInventoryCache(moduleManager, coreModuleConfig);
+
+        Mockito.when(moduleProvider.getService(IProfileThreadSnapshotQueryDAO.class)).thenReturn(new ProfileExportSnapshotDAO(exportedData));
+        Mockito.when(moduleProvider.getService(ITraceQueryDAO.class)).thenReturn(new ProfileTraceDAO(exportedData));
+        Mockito.when(moduleProvider.getService(ServiceInventoryCache.class)).thenReturn(serviceInventoryCache);
+    }
+
+    @Test
+    public void test() throws IOException {
+        final ExporterConfig config = new ExporterConfig();
+        config.setTraceId(exportedData.getTraceId());
+        config.setTaskId(exportedData.getTaskId());
+        config.setAnalyzeResultDist(new File("").getAbsolutePath());
+
+        // dump
+        final ProfiledBasicInfo basicInfo = ProfiledBasicInfo.build(config, moduleManager);
+        final File writeFile = ProfileSnapshotDumper.dump(basicInfo, moduleManager);
+        Assert.assertTrue(writeFile != null);
+        Assert.assertTrue(writeFile.exists());
+
+        // parse
+        final ProfileAnalyzeTimeRange timeRange = new ProfileAnalyzeTimeRange();
+        timeRange.setStart(exportedData.getSpans().get(0).getStart());
+        timeRange.setEnd(exportedData.getSpans().get(0).getEnd());
+        final List<ThreadSnapshot> threadSnapshots = ProfileSnapshotDumper.parseFromFileWithTimeRange(writeFile, Collections.singletonList(timeRange));
+
+        Assert.assertEquals(threadSnapshots.size(), exportedData.getSnapshots().size());
+        for (int i = 0; i < threadSnapshots.size(); i++) {
+            Assert.assertEquals(threadSnapshots.get(i).getSequence(), i);
+            Assert.assertEquals(threadSnapshots.get(i).getTime(), i * exportedData.getLimit());
+
+            final String[] snapshots = exportedData.getSnapshots().get(i).split("-");
+            for (int snapshotIndex = 0; snapshotIndex < snapshots.length; snapshotIndex++) {
+                Assert.assertEquals(threadSnapshots.get(i).getStack().getCodeSignaturesList().get(snapshotIndex), snapshots[snapshotIndex]);
+            }
+        }
+
+        writeFile.delete();
+    }
+
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-bootstrap/src/test/java/org/apache/skywalking/oap/server/tool/profile/exporter/test/ProfileTraceDAO.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.exporter.test;
+
+import org.apache.skywalking.apm.network.language.agent.v2.SegmentObject;
+import org.apache.skywalking.apm.network.language.agent.v2.SpanObjectV2;
+import org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentRecord;
+import org.apache.skywalking.oap.server.core.query.entity.QueryOrder;
+import org.apache.skywalking.oap.server.core.query.entity.Span;
+import org.apache.skywalking.oap.server.core.query.entity.TraceBrief;
+import org.apache.skywalking.oap.server.core.query.entity.TraceState;
+import org.apache.skywalking.oap.server.core.storage.query.ITraceQueryDAO;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class ProfileTraceDAO implements ITraceQueryDAO {
+    private final ExportedData exportData;
+
+    public ProfileTraceDAO(ExportedData exportData) {
+        this.exportData = exportData;
+    }
+
+    @Override
+    public TraceBrief queryBasicTraces(long startSecondTB, long endSecondTB, long minDuration, long maxDuration, String endpointName, int serviceId, int serviceInstanceId, int endpointId, String traceId, int limit, int from, TraceState traceState, QueryOrder queryOrder) throws IOException {
+        return null;
+    }
+
+    @Override
+    public List<SegmentRecord> queryByTraceId(String traceId) throws IOException {
+        final ArrayList<SegmentRecord> segments = new ArrayList<>();
+        final SegmentRecord segment = new SegmentRecord();
+        segments.add(segment);
+
+        final SegmentObject.Builder segmentBuilder = SegmentObject.newBuilder();
+        for (ExportedData.Span span : exportData.getSpans()) {
+            segmentBuilder.addSpans(SpanObjectV2.newBuilder()
+                    .setOperationName(span.getOperation())
+                    .setStartTime(span.getStart())
+                    .setEndTime(span.getEnd())
+                    .setSpanId(span.getId())
+                    .setParentSpanId(span.getParentId()));
+        }
+        segment.setDataBinary(segmentBuilder.build().toByteArray());
+        segment.setServiceId(1);
+        segment.setSegmentId(exportData.getSegmentId());
+        return segments;
+    }
+
+    @Override
+    public List<Span> doFlexibleTraceQuery(String traceId) throws IOException {
+        return null;
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-bootstrap/src/test/resources/profile.yml

+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+taskId: 1
+segmentId: 1.1.1
+traceId: 1.1.1
+limit: 10
+snapshots:
+  - a-b-c
+  - a-b-c
+  - a-b-c
+  - a-b-c
+spans:
+  - operation: a
+    start: 0
+    end: 40
+    id: 1
+    parentId: 0
+  - operation: b
+    start: 10
+    end: 30
+    id: 2
+    parentId: 1

oap-server/server-tools/profile-exporter/tool-profile-snapshot-exporter-es7/pom.xml

+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  ~
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>profile-exporter</artifactId>
+        <groupId>org.apache.skywalking</groupId>
+        <version>7.0.0-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <groupId>org.apache.skywalking</groupId>
+    <artifactId>tool-profile-snapshot-exporter-es7</artifactId>
+    <version>7.0.0-SNAPSHOT</version>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.skywalking</groupId>
+            <artifactId>tool-profile-snapshot-bootstrap</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.skywalking</groupId>
+            <artifactId>storage-elasticsearch7-plugin</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+    </dependencies>
+
+</project>
\ No newline at end of file

oap-server/server-tools/profile-exporter/tool-profile-snapshot-exporter-es7/src/main/java/org/apache/skywalking/oap/server/tool/profile/exporter/ProfileSnapshotExporter.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.exporter;
+
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ProfileSnapshotExporter {
+    public static void main(String[] args) {
+        ProfileSnapshotExporterBootstrap.export(args);
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-exporter/pom.xml

+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  ~
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>profile-exporter</artifactId>
+        <groupId>org.apache.skywalking</groupId>
+        <version>7.0.0-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <groupId>org.apache.skywalking</groupId>
+    <artifactId>tool-profile-snapshot-exporter</artifactId>
+    <version>7.0.0-SNAPSHOT</version>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.skywalking</groupId>
+            <artifactId>tool-profile-snapshot-bootstrap</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.skywalking</groupId>
+            <artifactId>storage-elasticsearch-plugin</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+    </dependencies>
+
+
+</project>
\ No newline at end of file

oap-server/server-tools/profile-exporter/tool-profile-snapshot-exporter/src/main/java/org/apache/skywalking/oap/server/tool/profile/exporter/ProfileSnapshotExporter.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.exporter;
+
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+public class ProfileSnapshotExporter {
+    public static void main(String[] args) {
+        ProfileSnapshotExporterBootstrap.export(args);
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-server-mock/pom.xml

+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  ~
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>profile-exporter</artifactId>
+        <groupId>org.apache.skywalking</groupId>
+        <version>7.0.0-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <groupId>org.apache.skywalking</groupId>
+    <artifactId>tool-profile-snapshot-server-mock</artifactId>
+    <version>7.0.0-SNAPSHOT</version>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.skywalking</groupId>
+            <artifactId>server-core</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+    </dependencies>
+
+</project>
\ No newline at end of file

oap-server/server-tools/profile-exporter/tool-profile-snapshot-server-mock/src/main/java/org/apache/skywalking/oap/server/tool/profile/core/MockCoreModuleConfig.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.core;
+
+import org.apache.skywalking.oap.server.library.module.ModuleConfig;
+
+public class MockCoreModuleConfig extends ModuleConfig {
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-server-mock/src/main/java/org/apache/skywalking/oap/server/tool/profile/core/MockCoreModuleProvider.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.core;
+
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.CoreModuleConfig;
+import org.apache.skywalking.oap.server.core.CoreModuleProvider;
+import org.apache.skywalking.oap.server.core.annotation.AnnotationScan;
+import org.apache.skywalking.oap.server.core.cache.EndpointInventoryCache;
+import org.apache.skywalking.oap.server.core.cache.NetworkAddressInventoryCache;
+import org.apache.skywalking.oap.server.core.cache.ProfileTaskCache;
+import org.apache.skywalking.oap.server.core.cache.ServiceInstanceInventoryCache;
+import org.apache.skywalking.oap.server.core.cache.ServiceInventoryCache;
+import org.apache.skywalking.oap.server.core.command.CommandService;
+import org.apache.skywalking.oap.server.core.config.ConfigService;
+import org.apache.skywalking.oap.server.core.config.DownsamplingConfigService;
+import org.apache.skywalking.oap.server.core.config.IComponentLibraryCatalogService;
+import org.apache.skywalking.oap.server.core.profile.ProfileTaskMutationService;
+import org.apache.skywalking.oap.server.core.query.AggregationQueryService;
+import org.apache.skywalking.oap.server.core.query.AlarmQueryService;
+import org.apache.skywalking.oap.server.core.query.LogQueryService;
+import org.apache.skywalking.oap.server.core.query.MetadataQueryService;
+import org.apache.skywalking.oap.server.core.query.MetricQueryService;
+import org.apache.skywalking.oap.server.core.query.ProfileTaskQueryService;
+import org.apache.skywalking.oap.server.core.query.TopNRecordsQueryService;
+import org.apache.skywalking.oap.server.core.query.TopologyQueryService;
+import org.apache.skywalking.oap.server.core.query.TraceQueryService;
+import org.apache.skywalking.oap.server.core.register.service.EndpointInventoryRegister;
+import org.apache.skywalking.oap.server.core.register.service.IEndpointInventoryRegister;
+import org.apache.skywalking.oap.server.core.register.service.INetworkAddressInventoryRegister;
+import org.apache.skywalking.oap.server.core.register.service.IServiceInstanceInventoryRegister;
+import org.apache.skywalking.oap.server.core.register.service.IServiceInventoryRegister;
+import org.apache.skywalking.oap.server.core.register.service.NetworkAddressInventoryRegister;
+import org.apache.skywalking.oap.server.core.register.service.ServiceInstanceInventoryRegister;
+import org.apache.skywalking.oap.server.core.register.service.ServiceInventoryRegister;
+import org.apache.skywalking.oap.server.core.remote.RemoteSenderService;
+import org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager;
+import org.apache.skywalking.oap.server.core.server.GRPCHandlerRegister;
+import org.apache.skywalking.oap.server.core.server.JettyHandlerRegister;
+import org.apache.skywalking.oap.server.core.source.DefaultScopeDefine;
+import org.apache.skywalking.oap.server.core.source.SourceReceiver;
+import org.apache.skywalking.oap.server.core.storage.model.IModelGetter;
+import org.apache.skywalking.oap.server.core.storage.model.IModelOverride;
+import org.apache.skywalking.oap.server.core.storage.model.IModelSetter;
+import org.apache.skywalking.oap.server.core.storage.model.StorageModels;
+import org.apache.skywalking.oap.server.core.worker.IWorkerInstanceGetter;
+import org.apache.skywalking.oap.server.core.worker.IWorkerInstanceSetter;
+import org.apache.skywalking.oap.server.library.module.ModuleConfig;
+import org.apache.skywalking.oap.server.library.module.ModuleDefine;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+import org.apache.skywalking.oap.server.library.module.ServiceNotProvidedException;
+import org.apache.skywalking.oap.server.telemetry.TelemetryModule;
+import org.apache.skywalking.oap.server.tool.profile.core.mock.MockComponentLibraryCatalogService;
+import org.apache.skywalking.oap.server.tool.profile.core.mock.MockGRPCHandlerRegister;
+import org.apache.skywalking.oap.server.tool.profile.core.mock.MockJettyHandlerRegister;
+import org.apache.skywalking.oap.server.tool.profile.core.mock.MockRemoteClientManager;
+import org.apache.skywalking.oap.server.tool.profile.core.mock.MockSourceReceiver;
+import org.apache.skywalking.oap.server.tool.profile.core.mock.MockStreamAnnotationListener;
+import org.apache.skywalking.oap.server.tool.profile.core.mock.MockWorkerInstancesService;
+
+import java.io.IOException;
+import java.util.Collections;
+
+public class MockCoreModuleProvider extends CoreModuleProvider {
+
+    private final StorageModels storageModels;
+    private final AnnotationScan annotationScan;
+
+    public MockCoreModuleProvider() {
+        this.storageModels = new StorageModels();
+        this.annotationScan = new AnnotationScan();
+    }
+
+    @Override
+    public String name() {
+        return "tool-profile-mock-core";
+    }
+
+    @Override
+    public Class<? extends ModuleDefine> module() {
+        return CoreModule.class;
+    }
+
+    @Override
+    public ModuleConfig createConfigBeanIfAbsent() {
+        return new MockCoreModuleConfig();
+    }
+
+    @Override
+    public void prepare() throws ServiceNotProvidedException, ModuleStartException {
+        MockStreamAnnotationListener streamAnnotationListener = new MockStreamAnnotationListener(getManager());
+        annotationScan.registerListener(streamAnnotationListener);
+
+        AnnotationScan scopeScan = new AnnotationScan();
+        scopeScan.registerListener(new DefaultScopeDefine.Listener());
+        try {
+            scopeScan.scan();
+        } catch (Exception e) {
+            throw new ModuleStartException(e.getMessage(), e);
+        }
+
+        CoreModuleConfig moduleConfig = new CoreModuleConfig();
+        this.registerServiceImplementation(ConfigService.class, new ConfigService(moduleConfig));
+        this.registerServiceImplementation(DownsamplingConfigService.class, new DownsamplingConfigService(Collections.emptyList()));
+
+        this.registerServiceImplementation(GRPCHandlerRegister.class, new MockGRPCHandlerRegister());
+        this.registerServiceImplementation(JettyHandlerRegister.class, new MockJettyHandlerRegister());
+
+        this.registerServiceImplementation(IComponentLibraryCatalogService.class, new MockComponentLibraryCatalogService());
+
+        this.registerServiceImplementation(SourceReceiver.class, new MockSourceReceiver());
+
+        MockWorkerInstancesService instancesService = new MockWorkerInstancesService();
+        this.registerServiceImplementation(IWorkerInstanceGetter.class, instancesService);
+        this.registerServiceImplementation(IWorkerInstanceSetter.class, instancesService);
+
+        this.registerServiceImplementation(RemoteSenderService.class, new RemoteSenderService(getManager()));
+        this.registerServiceImplementation(IModelSetter.class, storageModels);
+        this.registerServiceImplementation(IModelGetter.class, storageModels);
+        this.registerServiceImplementation(IModelOverride.class, storageModels);
+
+        this.registerServiceImplementation(ServiceInventoryCache.class, new ServiceInventoryCache(getManager(), moduleConfig));
+        this.registerServiceImplementation(IServiceInventoryRegister.class, new ServiceInventoryRegister(getManager()));
+
+        this.registerServiceImplementation(ServiceInstanceInventoryCache.class, new ServiceInstanceInventoryCache(getManager(), moduleConfig));
+        this.registerServiceImplementation(IServiceInstanceInventoryRegister.class, new ServiceInstanceInventoryRegister(getManager()));
+
+        this.registerServiceImplementation(EndpointInventoryCache.class, new EndpointInventoryCache(getManager(), moduleConfig));
+        this.registerServiceImplementation(IEndpointInventoryRegister.class, new EndpointInventoryRegister(getManager()));
+
+        this.registerServiceImplementation(NetworkAddressInventoryCache.class, new NetworkAddressInventoryCache(getManager(), moduleConfig));
+        this.registerServiceImplementation(INetworkAddressInventoryRegister.class, new NetworkAddressInventoryRegister(getManager()));
+
+        this.registerServiceImplementation(TopologyQueryService.class, new TopologyQueryService(getManager()));
+        this.registerServiceImplementation(MetricQueryService.class, new MetricQueryService(getManager()));
+        this.registerServiceImplementation(TraceQueryService.class, new TraceQueryService(getManager()));
+        this.registerServiceImplementation(LogQueryService.class, new LogQueryService(getManager()));
+        this.registerServiceImplementation(MetadataQueryService.class, new MetadataQueryService(getManager()));
+        this.registerServiceImplementation(AggregationQueryService.class, new AggregationQueryService(getManager()));
+        this.registerServiceImplementation(AlarmQueryService.class, new AlarmQueryService(getManager()));
+        this.registerServiceImplementation(TopNRecordsQueryService.class, new TopNRecordsQueryService(getManager()));
+
+        // add profile service implementations
+        this.registerServiceImplementation(
+            ProfileTaskMutationService.class, new ProfileTaskMutationService(getManager()));
+        this.registerServiceImplementation(
+            ProfileTaskQueryService.class, new ProfileTaskQueryService(getManager(), moduleConfig));
+        this.registerServiceImplementation(ProfileTaskCache.class, new ProfileTaskCache(getManager(), moduleConfig));
+
+        this.registerServiceImplementation(CommandService.class, new CommandService(getManager()));
+
+        this.registerServiceImplementation(RemoteClientManager.class, new MockRemoteClientManager(getManager(), 0));
+    }
+
+    @Override
+    public void start() throws ModuleStartException {
+        try {
+            annotationScan.scan();
+        } catch (IOException e) {
+            throw new ModuleStartException(e.getMessage(), e);
+        }
+    }
+
+    @Override
+    public void notifyAfterCompleted() {
+    }
+
+    @Override
+    public String[] requiredModules() {
+        return new String[] {
+                TelemetryModule.NAME
+        };
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-server-mock/src/main/java/org/apache/skywalking/oap/server/tool/profile/core/mock/MockComponentLibraryCatalogService.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.core.mock;
+
+import org.apache.skywalking.oap.server.core.config.IComponentLibraryCatalogService;
+
+/**
+ * Mock from {@link IComponentLibraryCatalogService}
+ */
+public class MockComponentLibraryCatalogService implements IComponentLibraryCatalogService {
+    @Override
+    public int getComponentId(String componentName) {
+        return 0;
+    }
+
+    @Override
+    public int getServerIdBasedOnComponent(int componentId) {
+        return 0;
+    }
+
+    @Override
+    public String getComponentName(int componentId) {
+        return null;
+    }
+
+    @Override
+    public String getServerNameBasedOnComponent(int componentId) {
+        return null;
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-server-mock/src/main/java/org/apache/skywalking/oap/server/tool/profile/core/mock/MockGRPCHandlerRegister.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.core.mock;
+
+import io.grpc.BindableService;
+import io.grpc.ServerInterceptor;
+import io.grpc.ServerServiceDefinition;
+import org.apache.skywalking.oap.server.core.server.GRPCHandlerRegister;
+
+/**
+ * Mock from {@link GRPCHandlerRegister}
+ */
+public class MockGRPCHandlerRegister implements GRPCHandlerRegister {
+    @Override
+    public void addHandler(BindableService handler) {
+    }
+
+    @Override
+    public void addHandler(ServerServiceDefinition definition) {
+    }
+
+    @Override
+    public void addFilter(ServerInterceptor interceptor) {
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-server-mock/src/main/java/org/apache/skywalking/oap/server/tool/profile/core/mock/MockJettyHandlerRegister.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.core.mock;
+
+import org.apache.skywalking.oap.server.core.server.JettyHandlerRegister;
+import org.apache.skywalking.oap.server.library.server.jetty.JettyHandler;
+
+/**
+ * Mock from {@link JettyHandlerRegister}
+ */
+public class MockJettyHandlerRegister implements JettyHandlerRegister {
+    @Override
+    public void addHandler(JettyHandler serverHandler) {
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-server-mock/src/main/java/org/apache/skywalking/oap/server/tool/profile/core/mock/MockRemoteClientManager.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.core.mock;
+
+import org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager;
+import org.apache.skywalking.oap.server.library.module.ModuleDefineHolder;
+
+/**
+ * Mock from {@link RemoteClientManager}
+ */
+public class MockRemoteClientManager extends RemoteClientManager {
+    public MockRemoteClientManager(ModuleDefineHolder moduleDefineHolder, int remoteTimeout) {
+        super(moduleDefineHolder, remoteTimeout);
+    }
+
+    @Override
+    public void start() {
+    }
+
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-server-mock/src/main/java/org/apache/skywalking/oap/server/tool/profile/core/mock/MockSourceReceiver.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.core.mock;
+
+import org.apache.skywalking.oap.server.core.source.Source;
+import org.apache.skywalking.oap.server.core.source.SourceReceiver;
+
+/**
+ * Mock from {@link SourceReceiver}
+ */
+public class MockSourceReceiver implements SourceReceiver {
+    @Override
+    public void receive(Source source) {
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-server-mock/src/main/java/org/apache/skywalking/oap/server/tool/profile/core/mock/MockStreamAnnotationListener.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.core.mock;
+
+import org.apache.skywalking.oap.server.core.UnexpectedException;
+import org.apache.skywalking.oap.server.core.analysis.Stream;
+import org.apache.skywalking.oap.server.core.analysis.StreamAnnotationListener;
+import org.apache.skywalking.oap.server.core.analysis.worker.NoneStreamingProcessor;
+import org.apache.skywalking.oap.server.core.analysis.worker.RecordStreamProcessor;
+import org.apache.skywalking.oap.server.core.annotation.AnnotationListener;
+import org.apache.skywalking.oap.server.core.register.worker.InventoryStreamProcessor;
+import org.apache.skywalking.oap.server.library.module.ModuleDefineHolder;
+
+import java.lang.annotation.Annotation;
+
+/**
+ * Mock from {@link StreamAnnotationListener}
+ */
+public class MockStreamAnnotationListener implements AnnotationListener {
+
+    private final ModuleDefineHolder moduleDefineHolder;
+
+    public MockStreamAnnotationListener(ModuleDefineHolder moduleDefineHolder) {
+        this.moduleDefineHolder = moduleDefineHolder;
+    }
+
+    @Override
+    public Class<? extends Annotation> annotation() {
+        return Stream.class;
+    }
+
+    @Override
+    public void notify(Class aClass) {
+        if (aClass.isAnnotationPresent(Stream.class)) {
+            Stream stream = (Stream) aClass.getAnnotation(Stream.class);
+
+            // remove metrics and top N mock
+            if (stream.processor().equals(InventoryStreamProcessor.class)) {
+                InventoryStreamProcessor.getInstance().create(moduleDefineHolder, stream, aClass);
+            } else if (stream.processor().equals(RecordStreamProcessor.class)) {
+                RecordStreamProcessor.getInstance().create(moduleDefineHolder, stream, aClass);
+            } else if (stream.processor().equals(NoneStreamingProcessor.class)) {
+                NoneStreamingProcessor.getInstance().create(moduleDefineHolder, stream, aClass);
+            }
+        } else {
+            throw new UnexpectedException("Stream annotation listener could only parse the class present stream annotation.");
+        }
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-server-mock/src/main/java/org/apache/skywalking/oap/server/tool/profile/core/mock/MockWorkerInstancesService.java

+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.tool.profile.core.mock;
+
+import org.apache.skywalking.oap.server.core.remote.data.StreamData;
+import org.apache.skywalking.oap.server.core.worker.AbstractWorker;
+import org.apache.skywalking.oap.server.core.worker.IWorkerInstanceGetter;
+import org.apache.skywalking.oap.server.core.worker.IWorkerInstanceSetter;
+import org.apache.skywalking.oap.server.core.worker.RemoteHandleWorker;
+import org.apache.skywalking.oap.server.core.worker.WorkerInstancesService;
+
+/**
+ * Mock from {@link WorkerInstancesService}
+ */
+public class MockWorkerInstancesService implements IWorkerInstanceSetter, IWorkerInstanceGetter {
+    @Override
+    public RemoteHandleWorker get(String nextWorkerName) {
+        return null;
+    }
+
+    @Override
+    public void put(String remoteReceiverWorkName, AbstractWorker instance, Class<? extends StreamData> streamDataClass) {
+    }
+}

oap-server/server-tools/profile-exporter/tool-profile-snapshot-server-mock/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider

+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+org.apache.skywalking.oap.server.tool.profile.core.MockCoreModuleProvider

test/e2e/e2e-profile/e2e-profile-test-runner/pom.xml

@@ -153,12 +153,36 @@
                             <containerNamePattern>%a-%t-%i</containerNamePattern>
                             <imagePullPolicy>Always</imagePullPolicy>
                             <images>
+                                <image>
+                                    <name>openjdk:8-jre-buster</name>
+                                    <alias>${e2e.container.name.prefix}-datasource</alias>
+                                    <run>
+                                        <ports>
+                                            <port>h2.port:1521</port>
+                                        </ports>
+                                        <entrypoint>
+                                            <shell>bash /h2/h2-install.sh</shell>
+                                        </entrypoint>
+                                        <wait>
+                                            <log>TCP server running at tcp</log>
+                                            <time>300000</time>
+                                        </wait>
+                                        <volumes>
+                                            <bind>
+                                                <volume>${project.basedir}/src/docker/h2:/h2</volume>
+                                            </bind>
+                                        </volumes>
+                                    </run>
+                                </image>
                                 <image>
                                     <name>skyapm/e2e-container:${e2e.container.version}</name>
-                                    <alias>${e2e.container.name.prefix}</alias>
+                                    <alias>${e2e.container.name.prefix}-runner</alias>
                                     <run>
                                         <env>
                                             <STORAGE>h2</STORAGE>
+                                            <SW_STORAGE_H2_URL>
+                                                jdbc:h2:tcp://${e2e.container.name.prefix}-datasource:1521/skywalking-oap-db
+                                            </SW_STORAGE_H2_URL>
                                             <INSTRUMENTED_SERVICE_1>
                                                 ${provider.name}-${project.version}.jar
                                             </INSTRUMENTED_SERVICE_1>
@@ -173,6 +197,12 @@
                                             <port>+webapp.host:webapp.port:8081</port>
                                             <port>+service.host:service.port:9090</port>
                                         </ports>
+                                        <dependsOn>
+                                            <container>${e2e.container.name.prefix}-datasource</container>
+                                        </dependsOn>
+                                        <links>
+                                            <link>${e2e.container.name.prefix}-datasource</link>
+                                        </links>
                                         <volumes>
                                             <bind>
                                                 <volume>${sw.home}:/sw</volume>
@@ -216,7 +246,7 @@
                                 <image>
                                     <name>elastic/elasticsearch:${elasticsearch.version}</name>
                                     <alias>${e2e.container.name.prefix}-elasticsearch</alias>
-                                    <run>
+                                   <run>
                                         <ports>
                                             <port>es.port:9200</port>
                                         </ports>
@@ -302,6 +332,27 @@
                         <configuration>
                             <containerNamePattern>%a-%t-%i</containerNamePattern>
                             <images>
+                                <image>
+                                    <name>openjdk:8-jre-buster</name>
+                                    <alias>${e2e.container.name.prefix}-h2</alias>
+                                    <run>
+                                        <ports>
+                                            <port>h2.port:1521</port>
+                                        </ports>
+                                        <entrypoint>
+                                            <shell>bash /h2/h2-install.sh</shell>
+                                        </entrypoint>
+                                        <wait>
+                                            <log>TCP server running at tcp</log>
+                                            <time>300000</time>
+                                        </wait>
+                                        <volumes>
+                                            <bind>
+                                                <volume>${project.basedir}/src/docker/h2:/h2</volume>
+                                            </bind>
+                                        </volumes>
+                                    </run>
+                                </image>
                                 <image>
                                     <name>influxdb:${influxdb.version}</name>
                                     <alias>${e2e.container.name.prefix}-influxdb</alias>
@@ -324,6 +375,7 @@
                                     <run>
                                         <env>
                                             <STORAGE>influxdb</STORAGE>
+                                            <SW_STORAGE_METABASE_URL>jdbc:h2:tcp://${e2e.container.name.prefix}-h2:1521/skywalking-oap-db</SW_STORAGE_METABASE_URL>
                                             <SW_STORAGE_INFLUXDB_URL>http://${e2e.container.name.prefix}-influxdb:8086</SW_STORAGE_INFLUXDB_URL>
                                             <INSTRUMENTED_SERVICE_1>
                                                 ${provider.name}-${project.version}.jar
@@ -336,6 +388,7 @@
                                             </INSTRUMENTED_SERVICE_1_OPTS>
                                         </env>
                                         <dependsOn>
+                                            <container>${e2e.container.name.prefix}-h2</container>
                                             <container>${e2e.container.name.prefix}-influxdb</container>
                                         </dependsOn>
                                         <ports>
@@ -343,6 +396,7 @@
                                             <port>+service.host:service.port:9090</port>
                                         </ports>
                                         <links>
+                                            <link>${e2e.container.name.prefix}-h2</link>
                                             <link>${e2e.container.name.prefix}-influxdb</link>
                                         </links>
                                         <volumes>
@@ -352,6 +406,7 @@
                                                 <volume>${project.basedir}/src/docker/rc.d:/rc.d:ro</volume>
                                                 <volume>${project.basedir}/src/docker/adapt_storage.awk:/adapt_storage.awk</volume>
                                                 <volume>${project.basedir}/src/docker/profile_official_analysis.oal:/profile_official_analysis.oal</volume>
+                                                <volume>${project.basedir}/src/docker/h2:/h2</volume>
                                             </bind>
                                         </volumes>
                                         <wait>
@@ -381,6 +436,11 @@
                         <service.host>${service.host}</service.host>
                         <service.port>${service.port}</service.port>
                         <provider.name>${provider.name}</provider.name>
+                        <SW_STORAGE_ES_CLUSTER_NODES>localhost:${es.port}</SW_STORAGE_ES_CLUSTER_NODES>
+                        <SW_STORAGE_INFLUXDB_URL>http://localhost:${influxdb.port}</SW_STORAGE_INFLUXDB_URL>
+                        <SW_JDBC_URL>jdbc:mysql://localhost:${mysql.port}/swtest</SW_JDBC_URL>
+                        <SW_STORAGE_H2_URL>jdbc:h2:tcp://localhost:${h2.port}/skywalking-oap-db</SW_STORAGE_H2_URL>
+                        <SW_STORAGE_METABASE_URL>jdbc:h2:tcp://localhost:${h2.port}/skywalking-oap-db</SW_STORAGE_METABASE_URL>
                     </systemPropertyVariables>
                 </configuration>
                 <executions>

test/e2e/e2e-profile/e2e-profile-test-runner/src/docker/h2/h2-install.sh

+#!/usr/bin/env bash
+# Licensed to the SkyAPM under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Download and start H2 server
+H2_RELEASE_DATE=2017-06-10
+curl -L http://www.h2database.com/h2-$H2_RELEASE_DATE.zip -o /tmp/h2-$H2_RELEASE_DATE.zip
+[[ $? -ne 0 ]] && echo "Fail to download h2: ${H2_RELEASE_DATE}." && exit 1
+
+# unzip h2 and run it
+cd /tmp
+unzip h2-$H2_RELEASE_DATE.zip \
+  && rm -f h2-$H2_RELEASE_DATE.zip \
+  && mkdir -p /tmp/h2/data
+
+java -cp /tmp/h2/bin/h2*.jar org.h2.tools.Server -tcp -tcpAllowOthers -tcpPort 1521 -baseDir /tmp/h2/data

test/e2e/e2e-profile/e2e-profile-test-runner/src/test/java/org/apache/skywalking/e2e/ProfileVerificationITCase.java

@@ -18,14 +18,23 @@
 
 package org.apache.skywalking.e2e;
 
+import java.io.BufferedReader;
+import java.io.File;
 import java.io.IOException;
 import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.file.Files;
 import java.time.LocalDateTime;
 import java.time.ZoneOffset;
+import java.util.ArrayList;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import java.util.Properties;
 import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+import com.google.common.base.Charsets;
 import org.apache.skywalking.e2e.profile.ProfileClient;
 import org.apache.skywalking.e2e.profile.creation.ProfileTaskCreationRequest;
 import org.apache.skywalking.e2e.profile.creation.ProfileTaskCreationResult;
@@ -213,6 +222,61 @@ public class ProfileVerificationITCase {
             expectedInputStream, ProfileStackTreeMatcher.class);
         servicesMatcher.verify(analyzation.getData().getTrees().get(0));
 
+        // verify shell exporter
+        String swHome = System.getProperty("sw.home");
+        copyStorageConfig(swHome);
+        String exporterBin = swHome + File.separator + "tools" + File.separator + "profile-exporter" + File.separator + "profile_exporter.sh";
+        validateExporter(exporterBin, swHome, taskId, foundedTrace.getTraceIds().get(0));
+    }
+
+    private void copyStorageConfig(String swHome) throws IOException {
+        String runtimeConfigPath = swHome + File.separator + "config" + File.separator + "application.yml";
+        String toolConfigPath = swHome + File.separator + "tools" + File.separator + "profile-exporter" + File.separator + "application.yml";
+
+        LOGGER.info("ready to copy storage config, from:{}, to:{}", runtimeConfigPath, toolConfigPath);
+
+        // reading e2e test config
+        List<String> runtimeStorageConfigLines = new ArrayList<>();
+        boolean currentInStorageConfig = false;
+        for (String runtimeConfigLine : Files.readAllLines(new File(runtimeConfigPath).toPath())) {
+            if (!currentInStorageConfig) {
+                currentInStorageConfig = runtimeConfigLine.startsWith("storage:");
+            } else if (runtimeConfigLine.matches("^\\S+\\:$")) {
+                currentInStorageConfig = false;
+            } else if (!runtimeConfigLine.startsWith("#")) {
+                runtimeStorageConfigLines.add(runtimeConfigLine);
+            }
+        }
+        assertThat(runtimeStorageConfigLines).isNotEmpty();
+        LOGGER.info("current e2e test storage config:");
+        runtimeStorageConfigLines.add(0, "storage:");
+        for (String storageLine : runtimeStorageConfigLines) {
+            LOGGER.info(storageLine);
+        }
+        LOGGER.info("------------");
+
+        // copy storage to tools config file
+        List<String> newToolConfigLines = new ArrayList<>();
+        for (String runtimeConfigLine : Files.readAllLines(new File(toolConfigPath).toPath())) {
+            if (!currentInStorageConfig) {
+                currentInStorageConfig = runtimeConfigLine.startsWith("storage:");
+            } else if (runtimeConfigLine.matches("^\\S+\\:$")) {
+                currentInStorageConfig = false;
+            }
+
+            if (!currentInStorageConfig) {
+                newToolConfigLines.add(runtimeConfigLine);
+            }
+        }
+        newToolConfigLines.addAll(runtimeStorageConfigLines);
+        LOGGER.info("copy to tools config file content:");
+        for (String storageLine : newToolConfigLines) {
+            LOGGER.info(storageLine);
+        }
+        LOGGER.info("------------");
+
+        // write new config content
+        Files.write(new File(toolConfigPath).toPath(), newToolConfigLines, Charsets.UTF_8);
     }
 
     private void verifyProfileTask(int serviceId, String verifyResources) throws InterruptedException {
@@ -298,4 +362,32 @@ public class ProfileVerificationITCase {
         }
     }
 
+    private void validateExporter(String exporterBin, String exportTo, String taskId, String traceId) throws IOException, InterruptedException {
+        String executeShell = exporterBin;
+        executeShell += " --taskid=" + taskId;
+        executeShell += " --traceid=" + traceId;
+        executeShell += " " + exportTo;
+
+        LOGGER.info("executing shell: {}", executeShell);
+
+        Properties properties = System.getProperties();
+        List<String> env = properties.entrySet().stream().map(e -> e.getKey() + "=" + e.getValue()).collect(Collectors.toList());
+        Process exec = Runtime.getRuntime().exec(executeShell, env.toArray(new String[env.size()]));
+
+        // print data
+        BufferedReader strCon = new BufferedReader(new InputStreamReader(exec.getInputStream()));
+        String line;
+        while ((line = strCon.readLine()) != null) {
+            LOGGER.info("executing: {}", line);
+        }
+        exec.waitFor();
+
+        // reading result file
+        File zipFile = new File(exportTo + File.separator + traceId + ".tar.gz");
+        assertThat(zipFile).canRead();
+
+        // delete it
+        zipFile.delete();
+    }
+
 }

tools/profile-exporter/application.yml

+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+core:
+  tool-profile-mock-core:
+storage:
+  #  elasticsearch:
+  #    nameSpace: ${SW_NAMESPACE:""}
+  #    clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:localhost:9200}
+  #    protocol: ${SW_STORAGE_ES_HTTP_PROTOCOL:"http"}
+  #    #trustStorePath: ${SW_SW_STORAGE_ES_SSL_JKS_PATH:"../es_keystore.jks"}
+  #    #trustStorePass: ${SW_SW_STORAGE_ES_SSL_JKS_PASS:""}
+  #    user: ${SW_ES_USER:""}
+  #    password: ${SW_ES_PASSWORD:""}
+  #    enablePackedDownsampling: ${SW_STORAGE_ENABLE_PACKED_DOWNSAMPLING:true} # Hour and Day metrics will be merged into minute index.
+  #    dayStep: ${SW_STORAGE_DAY_STEP:1} # Represent the number of days in the one minute/hour/day index.
+  #    indexShardsNumber: ${SW_STORAGE_ES_INDEX_SHARDS_NUMBER:2}
+  #    indexReplicasNumber: ${SW_STORAGE_ES_INDEX_REPLICAS_NUMBER:0}
+  #    # Those data TTL settings will override the same settings in core module.
+  #    recordDataTTL: ${SW_STORAGE_ES_RECORD_DATA_TTL:7} # Unit is day
+  #    otherMetricsDataTTL: ${SW_STORAGE_ES_OTHER_METRIC_DATA_TTL:45} # Unit is day
+  #    monthMetricsDataTTL: ${SW_STORAGE_ES_MONTH_METRIC_DATA_TTL:18} # Unit is month
+  #    # Batch process setting, refer to https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.5/java-docs-bulk-processor.html
+  #    bulkActions: ${SW_STORAGE_ES_BULK_ACTIONS:1000} # Execute the bulk every 1000 requests
+  #    flushInterval: ${SW_STORAGE_ES_FLUSH_INTERVAL:10} # flush the bulk every 10 seconds whatever the number of requests
+  #    concurrentRequests: ${SW_STORAGE_ES_CONCURRENT_REQUESTS:2} # the number of concurrent requests
+  #    resultWindowMaxSize: ${SW_STORAGE_ES_QUERY_MAX_WINDOW_SIZE:10000}
+  #    metadataQueryMaxSize: ${SW_STORAGE_ES_QUERY_MAX_SIZE:5000}
+  #    segmentQueryMaxSize: ${SW_STORAGE_ES_QUERY_SEGMENT_SIZE:200}
+  #    profileTaskQueryMaxSize: ${SW_STORAGE_ES_QUERY_PROFILE_TASK_SIZE:200}
+  #    advanced: ${SW_STORAGE_ES_ADVANCED:""}
+  elasticsearch7:
+    nameSpace: ${SW_NAMESPACE:""}
+    clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:localhost:9200}
+    protocol: ${SW_STORAGE_ES_HTTP_PROTOCOL:"http"}
+    #trustStorePath: ${SW_SW_STORAGE_ES_SSL_JKS_PATH:"../es_keystore.jks"}
+    #trustStorePass: ${SW_SW_STORAGE_ES_SSL_JKS_PASS:""}
+    enablePackedDownsampling: ${SW_STORAGE_ENABLE_PACKED_DOWNSAMPLING:true} # Hour and Day metrics will be merged into minute index.
+    dayStep: ${SW_STORAGE_DAY_STEP:1} # Represent the number of days in the one minute/hour/day index.
+    user: ${SW_ES_USER:""}
+    password: ${SW_ES_PASSWORD:""}
+    indexShardsNumber: ${SW_STORAGE_ES_INDEX_SHARDS_NUMBER:2}
+    indexReplicasNumber: ${SW_STORAGE_ES_INDEX_REPLICAS_NUMBER:0}
+    # Those data TTL settings will override the same settings in core module.
+    recordDataTTL: ${SW_STORAGE_ES_RECORD_DATA_TTL:7} # Unit is day
+    otherMetricsDataTTL: ${SW_STORAGE_ES_OTHER_METRIC_DATA_TTL:45} # Unit is day
+    monthMetricsDataTTL: ${SW_STORAGE_ES_MONTH_METRIC_DATA_TTL:18} # Unit is month
+    # Batch process setting, refer to https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.5/java-docs-bulk-processor.html
+    bulkActions: ${SW_STORAGE_ES_BULK_ACTIONS:1000} # Execute the bulk every 1000 requests
+    flushInterval: ${SW_STORAGE_ES_FLUSH_INTERVAL:10} # flush the bulk every 10 seconds whatever the number of requests
+    concurrentRequests: ${SW_STORAGE_ES_CONCURRENT_REQUESTS:2} # the number of concurrent requests
+    resultWindowMaxSize: ${SW_STORAGE_ES_QUERY_MAX_WINDOW_SIZE:10000}
+    metadataQueryMaxSize: ${SW_STORAGE_ES_QUERY_MAX_SIZE:5000}
+    segmentQueryMaxSize: ${SW_STORAGE_ES_QUERY_SEGMENT_SIZE:200}
+    advanced: ${SW_STORAGE_ES_ADVANCED:""}
+#  h2:
+#    driver: ${SW_STORAGE_H2_DRIVER:org.h2.jdbcx.JdbcDataSource}
+#    url: ${SW_STORAGE_H2_URL:jdbc:h2:mem:skywalking-oap-db}
+#    user: ${SW_STORAGE_H2_USER:sa}
+#    metadataQueryMaxSize: ${SW_STORAGE_H2_QUERY_MAX_SIZE:5000}
+#  mysql:
+#    properties:
+#      jdbcUrl: ${SW_JDBC_URL:"jdbc:mysql://localhost:3306/swtest"}
+#      dataSource.user: ${SW_DATA_SOURCE_USER:root}
+#      dataSource.password: ${SW_DATA_SOURCE_PASSWORD:root@1234}
+#      dataSource.cachePrepStmts: ${SW_DATA_SOURCE_CACHE_PREP_STMTS:true}
+#      dataSource.prepStmtCacheSize: ${SW_DATA_SOURCE_PREP_STMT_CACHE_SQL_SIZE:250}
+#      dataSource.prepStmtCacheSqlLimit: ${SW_DATA_SOURCE_PREP_STMT_CACHE_SQL_LIMIT:2048}
+#      dataSource.useServerPrepStmts: ${SW_DATA_SOURCE_USE_SERVER_PREP_STMTS:true}
+#    metadataQueryMaxSize: ${SW_STORAGE_MYSQL_QUERY_MAX_SIZE:5000}
+#  influx:
+#    # Metadata storage provider configuration
+#    metabaseType: ${SW_STORAGE_METABASE_TYPE:H2} # There are 2 options as Metabase provider, H2 or MySQL.
+#    h2Props:
+#      dataSourceClassName: ${SW_STORAGE_METABASE_DRIVER:org.h2.jdbcx.JdbcDataSource}
+#      dataSource.url: ${SW_STORAGE_METABASE_URL:jdbc:h2:mem:skywalking-oap-db}
+#      dataSource.user: ${SW_STORAGE_METABASE_USER:sa}
+#      dataSource.password: ${SW_STORAGE_METABASE_PASSWORD:}
+#    mysqlProps:
+#      jdbcUrl: ${SW_STORAGE_METABASE_URL:"jdbc:mysql://localhost:3306/swtest"}
+#      dataSource.user: ${SW_STORAGE_METABASE_USER:root}
+#      dataSource.password: ${SW_STORAGE_METABASE_PASSWORD:root@1234}
+#      dataSource.cachePrepStmts: ${SW_STORAGE_METABASE_CACHE_PREP_STMTS:true}
+#      dataSource.prepStmtCacheSize: ${SW_STORAGE_METABASE_PREP_STMT_CACHE_SQL_SIZE:250}
+#      dataSource.prepStmtCacheSqlLimit: ${SW_STORAGE_METABASE_PREP_STMT_CACHE_SQL_LIMIT:2048}
+#      dataSource.useServerPrepStmts: ${SW_STORAGE_METABASE_USE_SERVER_PREP_STMTS:true}
+#    metadataQueryMaxSize: ${SW_STORAGE_METABASE_QUERY_MAX_SIZE:5000}
+#    # InfluxDB configuration
+#    url: ${SW_STORAGE_INFLUXDB_URL:http://localhost:8086}
+#    user: ${SW_STORAGE_INFLUXDB_USER:root}
+#    password: ${SW_STORAGE_INFLUXDB_PASSWORD:}
+#    database: ${SW_STORAGE_INFLUXDB_DATABASE:skywalking}
+#    actions: ${SW_STORAGE_INFLUXDB_ACTIONS:1000} # the number of actions to collect
+#    duration: ${SW_STORAGE_INFLUXDB_DURATION:1000} # the time to wait at most (milliseconds)
+#    fetchTaskLogMaxSize: ${SW_STORAGE_INFLUXDB_FETCH_TASK_LOG_MAX_SIZE:5000} # the max number of fetch task log in a request
+telemetry:
+  none:

tools/profile-exporter/profile_exporter.sh

+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+bin_path=$0
+exporter_dir=$(cd $(dirname $0); pwd)
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --taskid=*)
+      task_id=${1#*=}
+      ;;
+    --traceid=*)
+      trace_id=${1#*=}
+      ;;
+    *)
+      export_path=$1
+  esac
+  shift
+done
+
+[[ ! ${task_id} || ! ${trace_id} || ! ${export_path} ]] \
+  && echo 'Usage: sh tools/profile-exporter/profile_exporter.sh [--taskid] [--traceid] export_path' \
+  && exit 1
+
+[[ ! -d ${export_path} ]] \
+  && echo "Cannot find export export_path path: ${export_path}" \
+  && exit 1
+
+# prepare paths
+oap_libs_dir="${exporter_dir}/../../oap-libs"
+exporter_log_file="${exporter_dir}/profile_exporter_log4j2.xml"
+tool_application_config="${exporter_dir}/application.yml"
+[[ ! -f ${tool_application_config} ]] \
+  && echo "Cannot find oap application.yml" \
+  && exit 1
+[[ ! -d ${oap_libs_dir} ]] \
+  && echo "Cannot find oap libs path" \
+  && exit 1
+
+# create current trace temporary path
+work_dir="${export_path}/${trace_id}"
+mkdir -p ${work_dir}
+
+# prepare exporter files
+mkdir -p "${work_dir}/config"
+mkdir -p "${work_dir}/work"
+cp ${exporter_log_file} ${work_dir}/config/log4j2.xml
+# only persist core and storage module in application.yml config
+cp ${tool_application_config} ${work_dir}/config/application.yml
+
+# start export
+echo "Exporting task: ${task_id}, trace: ${trace_id}, export_path: ${work_dir}"
+JAVA_OPTS=" -Xms256M -Xmx512M"
+_RUNJAVA=${JAVA_HOME}/bin/java
+[ -z "$JAVA_HOME" ] && _RUNJAVA=java
+
+CLASSPATH="${work_dir}/config:$CLASSPATH"
+for i in "${oap_libs_dir}"/*.jar
+do
+    CLASSPATH="$i:$CLASSPATH"
+done
+
+exec $_RUNJAVA ${JAVA_OPTS} -classpath $CLASSPATH org.apache.skywalking.oap.server.tool.profile.exporter.ProfileSnapshotExporter \
+  ${task_id} ${trace_id} ${work_dir}/work &
+wait
+
+if [ `ls -l ${work_dir}/work | wc -l` -lt 2 ]; then
+	echo "Export failure!"
+	exit 1
+fi
+
+# compress files(only compress work data, no config)
+echo "Compressing exported directory: ${work_dir}"
+CURRENT_DIR="$(cd "$(dirname $0)"; pwd)"
+cd ${work_dir}
+tar zcvf "${trace_id}.tar.gz" "./work"
+mv "${trace_id}.tar.gz" "../"
+cd $CURRENT_DIR
+
+# clear work files
+rm -rf "${work_dir}"
+
+echo "Profile export finished: ${work_dir}.tar.gz"

tools/profile-exporter/profile_exporter_log4j2.xml

+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  ~
+  -->
+
+<Configuration status="INFO">
+    <Appenders>
+        <Console name="Console" target="SYSTEM_OUT">
+            <PatternLayout charset="UTF-8" pattern="%d - %c - %L [%t] %-5p %x - %m%n"/>
+        </Console>
+    </Appenders>
+    <Loggers>
+        <Root level="INFO">
+            <AppenderRef ref="Console"/>
+        </Root>
+    </Loggers>
+</Configuration>