diff --git a/docs/documentation/cn/developer-guide/debug-tool.md b/docs/documentation/cn/developer-guide/debug-tool.md new file mode 100644 index 0000000000..5be5582975 --- /dev/null +++ b/docs/documentation/cn/developer-guide/debug-tool.md @@ -0,0 +1,247 @@ +# 调试工具 + +在Doris的使用、开发过程中，经常会遇到需要对Doris进行调试的场景，这里介绍一些常用的调试工具。 + +## 环境准备 + +[pprof](https://github.com/google/pprof): 来自gperftools，用于将gperftools所产生的内容转化成便于人可以阅读的格式，比如pdf, svg, text等. + +[graphviz](http://www.graphviz.org/): 在没有这个库的时候pprof只可以转化为text格式，但这种方式不易查看。那么安装这个库后，pprof可以转化为svg、pdf等格式，对于调用关系则更加清晰明了。 + +[perf](https://perf.wiki.kernel.org/index.php/Main_Page): linux内核自带性能分析工具。[这里](http://www.brendangregg.com/perf.html)有一些perf的使用例子。 + +[FlameGraph](https://github.com/brendangregg/FlameGraph): 可视化工具，用于将perf的输出以火焰图的形式展示出来。 + +## 内存 + +对于内存的调试一般分为两个方面。一个是内存使用的总量是否合理，内存使用量过大一方面可能是由于系统存在内存泄露，另一方面可能是因为程序内存使用不当。其次就是是否存在内存越界、非法访问的问题，比如程序访问一个非法地址的内存，使用了未初始化内存等。对于内存方面的调试我们一般使用如下几种方式来进行问题追踪。 + +### 查看日志 + +当发现内存使用量过大的时候，我们可以先查看be.out日志，看看是否有大内存申请。由于Doris当前使用的TCMalloc管理内存，那么遇到大内存申请时，都会将申请的堆栈打印到be.out文件中，一般的表现形式如下： + +``` +tcmalloc: large alloc 1396277248 bytes == 0x3f3488000 @ 0x2af6f63 0x2c4095b 0x134d278 0x134bdcb 0x133d105 0x133d1d0 0x19930ed +``` + +这个表示在Doris BE在这个堆栈上尝试申请`1396277248 bytes`的内存。我们可以通过`addr2line`命令去把堆栈还原成我们能够看懂的信，具体的例子如下所示。 + +``` +$ addr2line -e lib/palo_be 0x2af6f63 0x2c4095b 0x134d278 0x134bdcb 0x133d105 0x133d1d0 0x19930ed + +/home/ssd0/zc/palo/doris/core/thirdparty/src/gperftools-gperftools-2.7/src/tcmalloc.cc:1335 +/home/ssd0/zc/palo/doris/core/thirdparty/src/gperftools-gperftools-2.7/src/tcmalloc.cc:1357 +/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_table.cpp:267 +/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_table.hpp:86 +/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_join_node.cpp:239 +/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_join_node.cpp:213 +thread.cpp:? +``` + +### HEAP PROFILE + +有时内存的申请并不是大内存的申请导致，而是通过小内存不断的堆积导致的。那么就没有办法通过查看日志定位到具体的申请信息，那么就需要通过其他方式来获得信息。 + +这个时候我们可以利用TCMalloc的[HEAPPROFILE](https://gperftools.github.io/gperftools/heapprofile.html)的功能。如果设置了HEAPPROFILE功能，那么我们可以获得进程整体的内存申请使用情况。使用方式是在启动Doris BE前设置`HEAPPROFILE`环境变量。比如： + +``` +export HEAPPROFILE=/tmp/doris_be.hprof +./bin/start_be.sh --daemon +``` + +这样，当满足HEAPPROFILE的dump条件时，就会将内存的整体使用情况写到指定路径的文件中。后续我们就可以通过使用`pprof`工具来对输出的内容进行分析。 + +``` +$ pprof --text lib/palo_be /tmp/doris_be.hprof.0012.heap | head -30 + +Using local file lib/palo_be. +Using local file /tmp/doris_be.hprof.0012.heap. +Total: 668.6 MB + 610.6 91.3% 91.3% 610.6 91.3% doris::SystemAllocator::allocate_via_malloc (inline) + 18.1 2.7% 94.0% 18.1 2.7% _objalloc_alloc + 5.6 0.8% 94.9% 63.4 9.5% doris::RowBatch::RowBatch + 5.1 0.8% 95.6% 7.1 1.1% butil::ResourcePool::add_block (inline) + 3.7 0.5% 96.2% 3.7 0.5% butil::iobuf::create_block (inline) + 3.4 0.5% 96.7% 3.4 0.5% butil::FlatMap::init + 3.2 0.5% 97.2% 5.2 0.8% butil::ObjectPool::add_block (inline) + 2.6 0.4% 97.6% 2.6 0.4% __gnu_cxx::new_allocator::allocate (inline) + 2.0 0.3% 97.9% 2.0 0.3% butil::ObjectPool::add_block_group (inline) + 2.0 0.3% 98.2% 2.0 0.3% butil::ResourcePool::add_block_group (inline) + 1.7 0.3% 98.4% 1.7 0.3% doris::SegmentReader::_load_index +``` + +上述文件各个列的内容： + +* 第一列：函数直接申请的内存大小，单位MB +* 第四列：函数以及函数所有调用的函数总共内存大小。 +* 第二列、第五列分别是第一列与第四列的的比例值。 +* 第三列是个第二列的累积值。 + +当然也可以生成调用关系图片，更加方便分析。比如下面的命令就能够生成SVG格式的调用关系图。 + +``` +pprof --svg lib/palo_be /tmp/doris_be.hprof.0012.heap > heap.svg +``` + +**注意：开启这个选项是要影响程序的执行性能的，请慎重对线上的实例开启** + +### pprof remote server + +HEAPPROFILE虽然能够获得全部的内存使用信息，但是也有比较受限的地方。1. 需要重启BE进行。2. 需要一直开启这个命令，导致对整个进程的性能造成影响。 + +对Doris BE也可以使用动态开启、关闭heap profile的方式来对进程进行内存申请分析。Doris内部支持了GPerftools的[远程server调试](https://gperftools.github.io/gperftools/pprof_remote_servers.html)。那么可以通过`pprof`直接对远程运行的Doris BE进行动态的HEAP PROFILE。比如我们可以通过以下命令来查看Doris的内存的使用增量 + +``` +$ pprof --text --seconds=60 http://be_host:be_webport/pprof/heap + +Total: 1296.4 MB + 484.9 37.4% 37.4% 484.9 37.4% doris::StorageByteBuffer::create + 272.2 21.0% 58.4% 273.3 21.1% doris::RowBlock::init + 157.5 12.1% 70.5% 157.5 12.1% doris::RowBatch::RowBatch + 90.7 7.0% 77.5% 90.7 7.0% doris::SystemAllocator::allocate_via_malloc + 66.6 5.1% 82.7% 66.6 5.1% doris::IntegerColumnReader::init + 47.9 3.7% 86.4% 47.9 3.7% __gnu_cxx::new_allocator::allocate + 20.8 1.6% 88.0% 35.4 2.7% doris::SegmentReader::_load_index + 12.7 1.0% 89.0% 12.7 1.0% doris::DecimalColumnReader::init + 12.7 1.0% 89.9% 12.7 1.0% doris::LargeIntColumnReader::init + 12.7 1.0% 90.9% 12.7 1.0% doris::StringColumnDirectReader::init + 12.3 0.9% 91.9% 12.3 0.9% std::__cxx11::basic_string::_M_mutate + 10.4 0.8% 92.7% 10.4 0.8% doris::VectorizedRowBatch::VectorizedRowBatch + 10.0 0.8% 93.4% 10.0 0.8% doris::PlainTextLineReader::PlainTextLineReader +``` + +这个命令的输出与HEAP PROFILE的输出及查看方式一样，这里就不再详细说明。这个命令只有在执行的过程中才会开启统计，相比HEAP PROFILE对于进程性能的影响有限。 + +### LSAN + +[LSAN](https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer)是一个地址检查工具，GCC已经集成。在我们编译代码的时候开启相应的编译选项，就能够开启这个功能。当程序发生可以确定的内存泄露时，会将泄露堆栈打印。Doris BE已经集成了这个工具，只需要在编译的时候使用如下的命令进行编译就能够生成带有内存泄露检测版本的BE二进制 + +``` +BUILD_TYPE=LSAN ./build.sh +``` + +当系统检测到内存泄露的时候，就会在be.out里面输出对应的信息。为了下面的演示，我们故意在代码中插入一段内存泄露代码。我们在`StorageEngine`的`open`函数中插入如下代码 + +``` + char* leak_buf = new char[1024]; + strcpy(leak_buf, "hello world"); + LOG(INFO) << leak_buf; +``` + +我们就在be.out中获得了如下的输出 + +``` +================================================================= +==24732==ERROR: LeakSanitizer: detected memory leaks + +Direct leak of 1024 byte(s) in 1 object(s) allocated from: + #0 0xd10586 in operator new[](unsigned long) ../../../../gcc-7.3.0/libsanitizer/lsan/lsan_interceptors.cc:164 + #1 0xe333a2 in doris::StorageEngine::open(doris::EngineOptions const&, doris::StorageEngine**) /home/ssd0/zc/palo/doris/core/be/src/olap/storage_engine.cpp:104 + #2 0xd3cc96 in main /home/ssd0/zc/palo/doris/core/be/src/service/doris_main.cpp:159 + #3 0x7f573b5eebd4 in __libc_start_main (/opt/compiler/gcc-4.8.2/lib64/libc.so.6+0x21bd4) + +SUMMARY: LeakSanitizer: 1024 byte(s) leaked in 1 allocation(s). +``` + +从上述的输出中，我们能看到有1024个字节被泄露了，并且打印出来了内存申请时的堆栈信息。 + +**注意：开启这个选项是要影响程序的执行性能的，请慎重对线上的实例开启** + +**注意：如果开启了LSAN开关的话，tcmalloc就会被自动关闭** + +### ASAN + +除了内存使用不合理、泄露以外。有的时候也会发生内存访问非法地址等错误。这个时候我们可以借助[ASAN](https://github.com/google/sanitizers/wiki/AddressSanitizer)来辅助我们找到问题的原因。与LSAN一样，ASAN也集成在了GCC中。Doris通过如下的方式进行编译就能够开启这个功能 + +``` +BUILD_TYPE=ASAN ./build.sh +``` + +执行编译生成的二进制文件，当检测工具发现有异常访问时，就会立即退出，并将非法访问的堆栈输出在be.out中。对于ASAN的输出与LSAN是一样的分析方法。这里我们也主动注入一个地址访问错误，来展示下具体的内容输出。我们仍然在`StorageEngine`的`open`函数中注入一段非法内存访问，具体的错误代码如下 + +``` + char* invalid_buf = new char[1024]; + for (int i = 0; i < 1025; ++i) { + invalid_buf[i] = i; + } + LOG(INFO) << invalid_buf; +``` + +然后我们就会在be.out中获得如下的输出 + +``` +================================================================= +==23284==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61900008bf80 at pc 0x00000129f56a bp 0x7fff546eed90 sp 0x7fff546eed88 +WRITE of size 1 at 0x61900008bf80 thread T0 + #0 0x129f569 in doris::StorageEngine::open(doris::EngineOptions const&, doris::StorageEngine**) /home/ssd0/zc/palo/doris/core/be/src/olap/storage_engine.cpp:106 + #1 0xe2c1e3 in main /home/ssd0/zc/palo/doris/core/be/src/service/doris_main.cpp:159 + #2 0x7fa5580fbbd4 in __libc_start_main (/opt/compiler/gcc-4.8.2/lib64/libc.so.6+0x21bd4) + #3 0xd30794 (/home/ssd0/zc/palo/doris/core/output3/be/lib/palo_be+0xd30794) + +0x61900008bf80 is located 0 bytes to the right of 1024-byte region [0x61900008bb80,0x61900008bf80) +allocated by thread T0 here: + #0 0xdeb040 in operator new[](unsigned long) ../../../../gcc-7.3.0/libsanitizer/asan/asan_new_delete.cc:82 + #1 0x129f50d in doris::StorageEngine::open(doris::EngineOptions const&, doris::StorageEngine**) /home/ssd0/zc/palo/doris/core/be/src/olap/storage_engine.cpp:104 + #2 0xe2c1e3 in main /home/ssd0/zc/palo/doris/core/be/src/service/doris_main.cpp:159 + #3 0x7fa5580fbbd4 in __libc_start_main (/opt/compiler/gcc-4.8.2/lib64/libc.so.6+0x21bd4) + +SUMMARY: AddressSanitizer: heap-buffer-overflow /home/ssd0/zc/palo/doris/core/be/src/olap/storage_engine.cpp:106 in doris::StorageEngine::open(doris::EngineOptions const&, doris::StorageEngine**) +``` + +从这段信息中该可以看到在`0x61900008bf80`这个地址我们尝试去写一个字节，但是这个地址是非法的。我们也可以看到 `[0x61900008bb80,0x61900008bf80)`这个地址的申请堆栈。 + +**注意：开启这个选项是要影响程序的执行性能的，请慎重对线上的实例开启** + +**注意：如果开启了ASAN开关的话，tcmalloc就会被自动关闭** + +另外，如果be.out中输出了堆栈信息，但是并没有函数符号，那么这个时候需要我们手动的处理下才能获得可读的堆栈信息。具体的处理方法需要借助一个脚本来解析ASAN的输出。这个时候我们需要使用[asan_symbolize](https://llvm.org/svn/llvm-project/compiler-rt/trunk/lib/asan/scripts/asan_symbolize.py)来帮忙解析下。具体的使用方式如下： + +``` +cat be.out | python asan_symbolize.py | c++filt +``` + +通过上述的命令，我们就能够获得可读的堆栈信息了。 + +## CPU + +当系统的CPU Idle很低的时候，说明系统的CPU已经成为了主要瓶颈，这个时候就需要分析一下当前的CPU使用情况。对于Doris的BE可以有如下两种方式来分析Doris的CPU瓶颈。 + +### pprof + +由于Doris内部已经集成了并兼容了GPerf的REST接口，那么用户可以通过`pprof`工具来分析远程的Doris BE。具体的使用方式如下： + +``` +pprof --svg --seconds=60 http://be_host:be_webport/pprof/profile > be.svg +``` + +这样就能够生成一张BE执行的CPU消耗图。 + +![CPU Pprof](../../../resources/images/cpu-pprof-demo.png) + +### perf + flamegragh + +这个是相当通用的一种CPU分析方式，相比于`pprof`，这中方式必须要求能够登陆到分析对象的物理机上。但是相比于pprof只能定时采点，perf是能够通过不同的事件来完成堆栈信息采集的。具体的的使用方式如下： + +``` +perf record -g -p be_pid -- sleep 60 +``` + +这条命令会统计60秒钟BE的CPU运行情况，并且生成perf.data。对于perf.data的分析，可以通过perf的命令来进行分析 + +``` +perf report +``` + +分析得到如下的图片 + +![Perf Report](../../../resources/images/perf-report-demo.png) + +来对生成的内容进行分析。当然也可以使用flamegragh完成可视化展示。 + +``` +perf script | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > be.svg +``` + +这样也会生成一张当时运行的CPU消耗图。 + +![CPU Flame](../../../resources/images/cpu-flame-demo.svg) diff --git a/docs/documentation/en/developer-guide/debug-tool.md b/docs/documentation/en/developer-guide/debug-tool.md new file mode 100644 index 0000000000..530506a043 --- /dev/null +++ b/docs/documentation/en/developer-guide/debug-tool.md @@ -0,0 +1,247 @@ +# Debug Tool + +In the process of using and developing Doris, we often encounter scenarios that need to debug Doris. Here are some common debugging tools. + +## Preparing + +[pprof] (https://github.com/google/pprof): from gperftools, it is used to transform the content generated by gperftools into a format that is easy for people to read, such as PDF, SVG, text, etc. + +[graphviz] (http://www.graphviz.org/): in the absence of this library, pprof can only be converted to text format, but this way is not easy to view. After the library is installed, pprof can be converted to SVG, PDF and other formats, and the call relationship is clearer. + +[perf] (https://perf.wiki.kernel.org/index.php/main_page): Linux kernel comes with performance analysis tool. [here] (http://www.brendangregg.com/perf.html) there are some examples of perf usage. + +[flamegraph] (https://github.com/brendangregg/flamegraph): a visualization tool used to show the output of perf in the form of flame graph. + +## Memory + +Debugging memory is generally divided into two aspects. One is whether the total amount of memory use is reasonable. On the one hand, the excessive amount of memory use may be due to memory leak in the system, on the other hand, it may be due to improper use of program memory. The second is whether there is a problem of memory overrun and illegal access, such as program access to memory with an illegal address, use of uninitialized memory, etc. For the debugging of memory, we usually use the following ways to track the problems. + +### Log + +When we find that the memory usage is too large, we can first check the be.out log to see if there is a large memory application. Because of the TCMalloc currently used by Doris to manage memory, when a large memory application is encountered, the stack of the application will be printed to the be.out file. The general form is as follows: + +``` +tcmalloc: large alloc 1396277248 bytes == 0x3f3488000 @ 0x2af6f63 0x2c4095b 0x134d278 0x134bdcb 0x133d105 0x133d1d0 0x19930ed +``` + +This indicates that Doris be is trying to apply memory of '1396277248 bytes' on this stack. We can use the 'addr2line' command to restore the stack to a letter that we can understand. The specific example is shown below. + +``` +$ addr2line -e lib/palo_be 0x2af6f63 0x2c4095b 0x134d278 0x134bdcb 0x133d105 0x133d1d0 0x19930ed + +/home/ssd0/zc/palo/doris/core/thirdparty/src/gperftools-gperftools-2.7/src/tcmalloc.cc:1335 +/home/ssd0/zc/palo/doris/core/thirdparty/src/gperftools-gperftools-2.7/src/tcmalloc.cc:1357 +/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_table.cpp:267 +/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_table.hpp:86 +/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_join_node.cpp:239 +/home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_join_node.cpp:213 +thread.cpp:? +``` + +### HEAP PROFILE + +Sometimes the application of memory is not caused by the application of large memory, but by the continuous accumulation of small memory. Then there is no way to locate the specific application information by viewing the log, so you need to get the information through other ways. + +At this time, we can take advantage of TCMalloc's [heapprofile](https://gperftools.github.io/gperftools/heapprofile.html). If the heapprofile function is set, we can get the overall memory application usage of the process. The usage is to set the 'heapprofile' environment variable before starting Doris be. For example: + +``` +export HEAPPROFILE=/tmp/doris_be.hprof +./bin/start_be.sh --daemon +``` + +In this way, when the dump condition of the heapprofile is met, the overall memory usage will be written to the file in the specified path. Later, we can use the 'pprof' tool to analyze the output content. + +``` +$ pprof --text lib/palo_be /tmp/doris_be.hprof.0012.heap | head -30 + +Using local file lib/palo_be. +Using local file /tmp/doris_be.hprof.0012.heap. +Total: 668.6 MB + 610.6 91.3% 91.3% 610.6 91.3% doris::SystemAllocator::allocate_via_malloc (inline) + 18.1 2.7% 94.0% 18.1 2.7% _objalloc_alloc + 5.6 0.8% 94.9% 63.4 9.5% doris::RowBatch::RowBatch + 5.1 0.8% 95.6% 7.1 1.1% butil::ResourcePool::add_block (inline) + 3.7 0.5% 96.2% 3.7 0.5% butil::iobuf::create_block (inline) + 3.4 0.5% 96.7% 3.4 0.5% butil::FlatMap::init + 3.2 0.5% 97.2% 5.2 0.8% butil::ObjectPool::add_block (inline) + 2.6 0.4% 97.6% 2.6 0.4% __gnu_cxx::new_allocator::allocate (inline) + 2.0 0.3% 97.9% 2.0 0.3% butil::ObjectPool::add_block_group (inline) + 2.0 0.3% 98.2% 2.0 0.3% butil::ResourcePool::add_block_group (inline) + 1.7 0.3% 98.4% 1.7 0.3% doris::SegmentReader::_load_index +``` + +Contents of each column of the above documents: + +* Column 1: the memory size directly applied by the function, in MB +* Column 4: the total memory size of the function and all the functions it calls. +* The second column and the fifth column are the proportion values of the first column and the fourth column respectively. +* The third column is the cumulative value of the second column. + +Of course, it can also generate call relation pictures, which is more convenient for analysis. For example, the following command can generate a call graph in SVG format. + +``` +pprof --svg lib/palo_be /tmp/doris_be.hprof.0012.heap > heap.svg +``` + +**NOTE: turning on this option will affect the execution performance of the program. Please be careful to turn on the online instance.** + +### pprof remote server + +Although heapprofile can get all the memory usage information, it has some limitations. 1. Restart be. 2. You need to enable this command all the time, which will affect the performance of the whole process. + +For Doris be, you can also use the way of opening and closing the heap profile dynamically to analyze the memory application of the process. Doris supports the [remote server debugging of gperftools](https://gperftools.github.io/gperftools/pprof_remote_servers.html). Then you can use 'pprof' to directly perform dynamic head profile on the remote running Doris be. For example, we can check the memory usage increment of Doris through the following command + +``` +$ pprof --text --seconds=60 http://be_host:be_webport/pprof/heap + +Total: 1296.4 MB + 484.9 37.4% 37.4% 484.9 37.4% doris::StorageByteBuffer::create + 272.2 21.0% 58.4% 273.3 21.1% doris::RowBlock::init + 157.5 12.1% 70.5% 157.5 12.1% doris::RowBatch::RowBatch + 90.7 7.0% 77.5% 90.7 7.0% doris::SystemAllocator::allocate_via_malloc + 66.6 5.1% 82.7% 66.6 5.1% doris::IntegerColumnReader::init + 47.9 3.7% 86.4% 47.9 3.7% __gnu_cxx::new_allocator::allocate + 20.8 1.6% 88.0% 35.4 2.7% doris::SegmentReader::_load_index + 12.7 1.0% 89.0% 12.7 1.0% doris::DecimalColumnReader::init + 12.7 1.0% 89.9% 12.7 1.0% doris::LargeIntColumnReader::init + 12.7 1.0% 90.9% 12.7 1.0% doris::StringColumnDirectReader::init + 12.3 0.9% 91.9% 12.3 0.9% std::__cxx11::basic_string::_M_mutate + 10.4 0.8% 92.7% 10.4 0.8% doris::VectorizedRowBatch::VectorizedRowBatch + 10.0 0.8% 93.4% 10.0 0.8% doris::PlainTextLineReader::PlainTextLineReader +``` + +The output of this command is the same as the output and view mode of heap profile, which will not be described in detail here. Statistics will be enabled only during execution of this command, which has a limited impact on process performance compared with heap profile. + +### LSAN + +[LSAN](https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer) is an address checking tool, GCC has been integrated. When we compile the code, we can enable this function by turning on the corresponding compilation options. When the program has a determinable memory leak, it prints the leak stack. Doris be has integrated this tool, only need to compile with the following command to generate be binary with memory leak detection version. + +``` +BUILD_TYPE=LSAN ./build.sh +``` + +When the system detects a memory leak, it will output the corresponding information in be. Out. For the following demonstration, we intentionally insert a memory leak code into the code. We insert the following code into the `open` function of `StorageEngine`. + +``` + char* leak_buf = new char[1024]; + strcpy(leak_buf, "hello world"); + LOG(INFO) << leak_buf; +``` + +We get the following output in be.out + +``` +================================================================= +==24732==ERROR: LeakSanitizer: detected memory leaks + +Direct leak of 1024 byte(s) in 1 object(s) allocated from: + #0 0xd10586 in operator new[](unsigned long) ../../../../gcc-7.3.0/libsanitizer/lsan/lsan_interceptors.cc:164 + #1 0xe333a2 in doris::StorageEngine::open(doris::EngineOptions const&, doris::StorageEngine**) /home/ssd0/zc/palo/doris/core/be/src/olap/storage_engine.cpp:104 + #2 0xd3cc96 in main /home/ssd0/zc/palo/doris/core/be/src/service/doris_main.cpp:159 + #3 0x7f573b5eebd4 in __libc_start_main (/opt/compiler/gcc-4.8.2/lib64/libc.so.6+0x21bd4) + +SUMMARY: LeakSanitizer: 1024 byte(s) leaked in 1 allocation(s). +``` + +From the above output, we can see that 1024 bytes have been leaked, and the stack information of memory application has been printed out. + +**NOTE: turning on this option will affect the execution performance of the program. Please be careful to turn on the online instance.** + +**NOTE: if the LSAN switch is turned on, the TCMalloc will be automatically turned off** + +### ASAN + +Except for the unreasonable use and leakage of memory. Sometimes there will be memory access illegal address and other errors. At this time, we can use [ASAN](https://github.com/google/sanitizers/wiki/addresssanitizer) to help us find the cause of the problem. Like LSAN, ASAN is integrated into GCC. Doris can open this function by compiling as follows + +``` +BUILD_TYPE=ASAN ./build.sh +``` + +Execute the binary generated by compilation. When the detection tool finds any abnormal access, it will immediately exit and output the stack illegally accessed in be.out. The output of ASAN is the same as that of LSAN. Here we also actively inject an address access error to show the specific content output. We still inject an illegal memory access into the 'open' function of 'storageengine'. The specific error code is as follows + +``` + char* invalid_buf = new char[1024]; + for (int i = 0; i < 1025; ++i) { + invalid_buf[i] = i; + } + LOG(INFO) << invalid_buf; +``` + +We get the following output in be.out + +``` +================================================================= +==23284==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61900008bf80 at pc 0x00000129f56a bp 0x7fff546eed90 sp 0x7fff546eed88 +WRITE of size 1 at 0x61900008bf80 thread T0 + #0 0x129f569 in doris::StorageEngine::open(doris::EngineOptions const&, doris::StorageEngine**) /home/ssd0/zc/palo/doris/core/be/src/olap/storage_engine.cpp:106 + #1 0xe2c1e3 in main /home/ssd0/zc/palo/doris/core/be/src/service/doris_main.cpp:159 + #2 0x7fa5580fbbd4 in __libc_start_main (/opt/compiler/gcc-4.8.2/lib64/libc.so.6+0x21bd4) + #3 0xd30794 (/home/ssd0/zc/palo/doris/core/output3/be/lib/palo_be+0xd30794) + +0x61900008bf80 is located 0 bytes to the right of 1024-byte region [0x61900008bb80,0x61900008bf80) +allocated by thread T0 here: + #0 0xdeb040 in operator new[](unsigned long) ../../../../gcc-7.3.0/libsanitizer/asan/asan_new_delete.cc:82 + #1 0x129f50d in doris::StorageEngine::open(doris::EngineOptions const&, doris::StorageEngine**) /home/ssd0/zc/palo/doris/core/be/src/olap/storage_engine.cpp:104 + #2 0xe2c1e3 in main /home/ssd0/zc/palo/doris/core/be/src/service/doris_main.cpp:159 + #3 0x7fa5580fbbd4 in __libc_start_main (/opt/compiler/gcc-4.8.2/lib64/libc.so.6+0x21bd4) + +SUMMARY: AddressSanitizer: heap-buffer-overflow /home/ssd0/zc/palo/doris/core/be/src/olap/storage_engine.cpp:106 in doris::StorageEngine::open(doris::EngineOptions const&, doris::StorageEngine**) +``` + +From this message, we can see that at the address of `0x61900008bf80`, we tried to write a byte, but this address is illegal. We can also see the application stack of the address `[0x61900008bb80, 0x61900008bf80]`. + +**NOTE: turning on this option will affect the execution performance of the program. Please be careful to turn on the online instance.** + +**NOTE: if the ASAN switch is turned on, the TCMalloc will be automatically turned off** + +In addition, if stack information is output in be.out, but there is no function symbol, then we need to handle it manually to get readable stack information. The specific processing method needs a script to parse the output of ASAN. At this time, we need to use [asan_symbolize](https://llvm.org/svn/llvm-project/compiler-rt/trunk/lib/asan/scripts/asan_symbolize.py) to help with parsing. The specific usage is as follows: + +``` +cat be.out | python asan_symbolize.py | c++filt +``` + +With the above command, we can get readable stack information. + +## CPU + +When the CPU idle of the system is very low, it means that the CPU of the system has become the main bottleneck. At this time, it is necessary to analyze the current CPU usage. For the be of Doris, there are two ways to analyze the CPU bottleneck of Doris. + +### pprof + +Because Doris has integrated and compatible with GPERF rest interface, users can analyze remote Doris be through the 'pprof' tool. The specific usage is as follows: + +``` +pprof --svg --seconds=60 http://be_host:be_webport/pprof/profile > be.svg +``` + +In this way, a CPU consumption graph of be execution can be generated. + +![CPU Pprof](../../../resources/images/cpu-pprof-demo.png) + +### perf + flamegragh + +This is a quite common CPU analysis method. Compared with `pprof`, this method must be able to log in to the physical machine of the analysis object. However, compared with pprof, which can only collect points on time, perf can collect stack information through different events. The specific usage is as follows: + +``` +perf record -g -p be_pid -- sleep 60 +``` + +This command counts the CPU operation of be for 60 seconds and generates perf.data. For the analysis of perf.data, the command of perf can be used for analysis. + +``` +perf report +``` + +The analysis results in the following pictures + +![Perf Report](../../../resources/images/perf-report-demo.png) + +To analyze the generated content. Of course, you can also use flash graph to complete the visual display. + +``` +perf script | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > be.svg +``` + +This will also generate a graph of CPU consumption at that time. + +![CPU Flame](../../../resources/images/cpu-flame-demo.svg) diff --git a/docs/resources/images/cpu-flame-demo.svg b/docs/resources/images/cpu-flame-demo.svg new file mode 100644 index 0000000000..f08a8a7605 --- /dev/null +++ b/docs/resources/images/cpu-flame-demo.svg @@ -0,0 +1,3587 @@ + + + + + + + + + + + + + +Flame Graph + +Reset Zoom +Search + + +doris::PInternalServiceImpl<doris::PBackendService>::exec_plan_fragment (128 samples, 0.08%) + + + +doris::FieldTypeTraits< (212 samples, 0.14%) + + + +doris::SegmentWriter::~SegmentWriter (35 samples, 0.02%) + + + +doris::RowBlock::~RowBlock (33 samples, 0.02%) + + + +sys_pread64 (29 samples, 0.02%) + + + +doris::ColumnData::_seek_to_block (15 samples, 0.01%) + + + +doris::SegmentReader::get_block (1,650 samples, 1.06%) + + + +void std::__adjust_heap<__gnu_cxx::__normal_iterator<doris::CollectIterator::ChildCtx**, std::vector<doris::CollectIterator::ChildCtx*, std::allocator<doris::CollectIterator::ChildCtx*> > >, long, doris::CollectIterator::ChildCtx*, __gnu_cxx::__ops::_Iter_comp_iter<doris::CollectIterator::ChildCtxComparator> > (990 samples, 0.64%) + + + +doris::RunLengthIntegerWriter::write (2,047 samples, 1.32%) + + + +doris::OutStream::_create_new_input_buffer (33 samples, 0.02%) + + + +thread_group_cputime (55 samples, 0.04%) + + + +doris::VarStringColumnReader<doris::StringColumnDirectReader>::next_vector (321 samples, 0.21%) + + + +intel_pmu_enable_all (60 samples, 0.04%) + + + +doris::ser::read_var_unsigned (37 samples, 0.02%) + + + +doris::FieldTypeTraits< (1,026 samples, 0.66%) + + + +find_next_bit (18 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_patched_base_values (786 samples, 0.51%) + + + +doris::TPlanNode::read (16 samples, 0.01%) + + + +doris::memory_copy (78 samples, 0.05%) + + + +doris::ReadOnlyFileStream::_assure_data (36 samples, 0.02%) + + + +doris::RunLengthIntegerReader::_read_values (20 samples, 0.01%) + + + +brpc::ProcessInputMessage (148 samples, 0.10%) + + + +bvar::LatencyRecorder::latency_percentile (19 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_direct_values (388 samples, 0.25%) + + + +doris::ReadOnlyFileStream::_assure_data (15 samples, 0.01%) + + + +thread_group_cputime (14 samples, 0.01%) + + + +doris::NewPartitionedAggregationNode::open (34 samples, 0.02%) + + + +doris::IntegerColumnReaderWrapper<int, true>::next_vector (149 samples, 0.10%) + + + +memset (17 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_values (34 samples, 0.02%) + + + +doris::RunLengthIntegerReader::_read_values (27 samples, 0.02%) + + + +doris::MasterServerClient::report (34 samples, 0.02%) + + + +doris::ColumnStatistics::attach (29 samples, 0.02%) + + + +audit_syscall_entry (47 samples, 0.03%) + + + +thread_group_cputime (17 samples, 0.01%) + + + +doris::SegmentReader::get_block (14 samples, 0.01%) + + + +doris::LargeIntColumnWriter::write_batch (91 samples, 0.06%) + + + +doris::StorageByteBuffer::put (25 samples, 0.02%) + + + +doris::VectorizedRowBatch::dump_to_row_block (51 samples, 0.03%) + + + +doris::SegmentReader::_set_column_map (22 samples, 0.01%) + + + +doris::ser::read_var_unsigned (135 samples, 0.09%) + + + +thread_group_cputime (49 samples, 0.03%) + + + +doris::RunLengthIntegerReader::_read_delta_values (41 samples, 0.03%) + + + +tcmalloc::ThreadCache::ReleaseToCentralCache (18 samples, 0.01%) + + + +doris::olap_crc32 (74 samples, 0.05%) + + + +copy_user_generic_string (1,027 samples, 0.66%) + + + +doris::FragmentMgr::exec_plan_fragment (84 samples, 0.05%) + + + +doris::olap_decompress (45 samples, 0.03%) + + + +doris::RunLengthIntegerReader::_read_values (86 samples, 0.06%) + + + +thread_proxy (44,046 samples, 28.41%) +thread_proxy + + +doris::ColumnData::prepare_block_read (32 samples, 0.02%) + + + +doris::IntegerColumnReaderWrapper<short, true>::next_vector (192 samples, 0.12%) + + + +doris::ColumnData::set_read_params (89 samples, 0.06%) + + + +thread_group_cputime (78 samples, 0.05%) + + + +default_send_IPI_mask_sequence_phys (35 samples, 0.02%) + + + +tcmalloc::CentralFreeList::InsertRange (17 samples, 0.01%) + + + +doris::OLAPTable::acquire_data_sources (54 samples, 0.03%) + + + +doris::ser::write_var_unsigned (19 samples, 0.01%) + + + +doris::ser::find_closet_num_bits (27 samples, 0.02%) + + + +doris::RunLengthIntegerReader::_read_delta_values (174 samples, 0.11%) + + + +doris::OutStream::write (63 samples, 0.04%) + + + +doris::FieldTypeTraits< (1,569 samples, 1.01%) + + + +doris::ReadOnlyFileStream::_assure_data (14 samples, 0.01%) + + + +doris::ColumnStatistics::merge (35 samples, 0.02%) + + + +sysret_check (15 samples, 0.01%) + + + +doris::memory_copy (680 samples, 0.44%) + + + +doris::RunLengthIntegerReader::_read_delta_values (53 samples, 0.03%) + + + +__alloc_pages_nodemask (17 samples, 0.01%) + + + +doris::IntegerColumnWriterWrapper<long, true>::write_batch (2,372 samples, 1.53%) + + + +doris::IntegerColumnReaderWrapper<int, true>::next_vector (52 samples, 0.03%) + + + +radix_tree_lookup_slot (133 samples, 0.09%) + + + +thread_group_cputime (74 samples, 0.05%) + + + +doris::ColumnData::~ColumnData (183 samples, 0.12%) + + + +doris::Reader::init (36 samples, 0.02%) + + + +doris::MemPool::free_all (14 samples, 0.01%) + + + +doris::ser::read_ints (57 samples, 0.04%) + + + +doris::Reader::_attach_data_to_merge_set (41,285 samples, 26.63%) +doris::Reader::_attach_data_to_merge_set + + +doris::OutStream::write (15 samples, 0.01%) + + + +doris::RunLengthIntegerWriter::_write_delta_values (107 samples, 0.07%) + + + +void std::__adjust_heap<__gnu_cxx::__normal_iterator<doris::CollectIterator::ChildCtx**, std::vector<doris::CollectIterator::ChildCtx*, std::allocator<doris::CollectIterator::ChildCtx*> > >, long, doris::CollectIterator::ChildCtx*, __gnu_cxx::__ops::_Iter_comp_iter<doris::CollectIterator::ChildCtxComparator> > (6,633 samples, 4.28%) +void .. + + +__GI___libc_write (93 samples, 0.06%) + + + +tcmalloc::CentralFreeList::FetchFromOneSpansSafe (18 samples, 0.01%) + + + +__GI___libc_nanosleep (82 samples, 0.05%) + + + +doris::ReadOnlyFileStream::_assure_data (19 samples, 0.01%) + + + +__might_sleep (22 samples, 0.01%) + + + +doris::RunLengthIntegerWriter::_prepare_patched_blob (164 samples, 0.11%) + + + + (15 samples, 0.01%) + + + +doris::RowCursor::allocate_memory_for_string_type (19 samples, 0.01%) + + + +doris::ReadOnlyFileStream::_assure_data (46 samples, 0.03%) + + + +doris::lz4_decompress (32 samples, 0.02%) + + + +doris::ReadOnlyFileStream::_assure_data (53 samples, 0.03%) + + + +doris::SegmentReader::_seek_to_block_directly (877 samples, 0.57%) + + + +doris::IntegerColumnReaderWrapper<short, true>::next_vector (235 samples, 0.15%) + + + +doris::SegmentReader::_load_to_vectorized_row_batch (1,645 samples, 1.06%) + + + +doris::ColumnDataWriter::attached_by (9,791 samples, 6.32%) +doris::C.. + + +doris::MemPool::~MemPool (36 samples, 0.02%) + + + +doris::ser::read_ints (20 samples, 0.01%) + + + +find_busiest_group (32 samples, 0.02%) + + + +doris::RowCursor::init (21 samples, 0.01%) + + + +thread_group_cputime (22 samples, 0.01%) + + + +doris::Reader::_attach_data_to_merge_set (33 samples, 0.02%) + + + +doris::memory_copy (21 samples, 0.01%) + + + +brpc::policy::ProcessRpcRequest (144 samples, 0.09%) + + + +doris::IntegerColumnReaderWrapper<long, true>::next_vector (2,963 samples, 1.91%) +d.. + + +doris::ser::find_closet_num_bits (23 samples, 0.01%) + + + +doris::OutStream::_spill (15 samples, 0.01%) + + + +doris::RunLengthIntegerWriter::write (85 samples, 0.05%) + + + +doris::SegmentReader::_seek_to_block_directly (50 samples, 0.03%) + + + +doris::FieldTypeTraits< (134 samples, 0.09%) + + + +doris::ReadOnlyFileStream::_assure_data (37 samples, 0.02%) + + + +doris::IntegerColumnReaderWrapper<short, true>::seek (301 samples, 0.19%) + + + +doris::RunLengthIntegerWriter::_write_direct_values (190 samples, 0.12%) + + + +fget_light (18 samples, 0.01%) + + + + (17 samples, 0.01%) + + + +tcmalloc::CentralFreeList::FetchFromOneSpansSafe (52 samples, 0.03%) + + + +memset (76 samples, 0.05%) + + + +base::internal::SpinLockWake (20 samples, 0.01%) + + + +doris::ser::read_ints (34 samples, 0.02%) + + + +doris::InListPredicate<doris::StringValue>::evaluate (967 samples, 0.62%) + + + +tcmalloc::CentralFreeList::RemoveRange (14 samples, 0.01%) + + + +doris::ReadOnlyFileStream::_assure_data (50 samples, 0.03%) + + + +doris::NewPartitionedAggregationNode::init (16 samples, 0.01%) + + + +doris::RunLengthIntegerWriter::_write_values (25 samples, 0.02%) + + + +intel_pmu_enable_all (288 samples, 0.19%) + + + +doris::StorageByteBuffer::put (31 samples, 0.02%) + + + +ext4_mark_iloc_dirty (29 samples, 0.02%) + + + +__madvise (291 samples, 0.19%) + + + +doris::MemPool::FindChunk (55 samples, 0.04%) + + + +tcmalloc::CentralFreeList::Populate (17 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_delta_values (14 samples, 0.01%) + + + +doris::StorageByteBuffer::create (68 samples, 0.04%) + + + +doris::SegmentReader::_create_reader (94 samples, 0.06%) + + + +doris::OutStream::write (18 samples, 0.01%) + + + +void std::__adjust_heap<__gnu_cxx::__normal_iterator<doris::CollectIterator::ChildCtx**, std::vector<doris::CollectIterator::ChildCtx*, std::allocator<doris::CollectIterator::ChildCtx*> > >, long, doris::CollectIterator::ChildCtx*, __gnu_cxx::__ops::_Iter_comp_iter<doris::CollectIterator::ChildCtxComparator> > (34 samples, 0.02%) + + + +syscall (14 samples, 0.01%) + + + +doris::PushHandler::_convert (215 samples, 0.14%) + + + +doris::RunLengthIntegerReader::_read_delta_values (488 samples, 0.31%) + + + +__GI___strncmp_ssse3 (70 samples, 0.05%) + + + +doris::ExecNode::create_tree (32 samples, 0.02%) + + + +bvar::detail::combine (16 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_delta_values (45 samples, 0.03%) + + + +doris::ser::read_ints (68 samples, 0.04%) + + + +doris::ColumnData::prepare_block_read (41,267 samples, 26.62%) +doris::ColumnData::prepare_block_read + + +doris::SegmentReader::_pick_row_groups (194 samples, 0.13%) + + + +start_thread (498 samples, 0.32%) + + + +doris::OutStream::write (18 samples, 0.01%) + + + +doris::ColumnData::init (23 samples, 0.01%) + + + +doris::DoubleColumnWriterBase<double>::write_batch (742 samples, 0.48%) + + + +thread_group_cputime (20 samples, 0.01%) + + + +doris::OutStream::get_total_buffer_size (19 samples, 0.01%) + + + +tcmalloc::ThreadCache::FetchFromCentralCache (22 samples, 0.01%) + + + +tcmalloc::CentralFreeList::ReleaseToSpans (16 samples, 0.01%) + + + +doris::OutStream::get_total_buffer_size (1,785 samples, 1.15%) + + + +doris::FieldTypeTraits< (3,076 samples, 1.98%) +d.. + + +doris::SegmentWriter::finalize (206 samples, 0.13%) + + + +doris::IntegerColumnReaderWrapper<int, true>::seek (70 samples, 0.05%) + + + +doris::Reader::_agg_key_next_row (7,814 samples, 5.04%) +doris:.. + + +doris::RunLengthIntegerReader::_read_values (60 samples, 0.04%) + + + +doris::StorageByteBuffer::put (61 samples, 0.04%) + + + +doris::RowCursor::index_cmp (54 samples, 0.03%) + + + +doris::FragmentExecState::execute (61 samples, 0.04%) + + + +tcmalloc::CentralFreeList::RemoveRange (59 samples, 0.04%) + + + +doris::StreamIndexWriter::add_index_entry (55 samples, 0.04%) + + + +doris::ReadOnlyFileStream::_assure_data (18 samples, 0.01%) + + + +rw_verify_area (26 samples, 0.02%) + + + +base::internal::SpinLockDelay (32 samples, 0.02%) + + + +tcmalloc::CentralFreeList::RemoveRange (22 samples, 0.01%) + + + +doris::ser::read_ints (641 samples, 0.41%) + + + +doris::FieldTypeTraits< (501 samples, 0.32%) + + + +std::vector<doris::TTabletInfo, std::allocator<doris::TTabletInfo> >::operator= (16 samples, 0.01%) + + + +doris::PositionEntryReader::positions (18 samples, 0.01%) + + + +tcmalloc::CentralFreeList::ReleaseListToSpans (26 samples, 0.02%) + + + +bvar::detail::Percentile::reset (15 samples, 0.01%) + + + +doris::SegmentWriter::_make_file_header (40 samples, 0.03%) + + + +doris::ReadOnlyFileStream::_assure_data (72 samples, 0.05%) + + + +doris::ser::write_ints (133 samples, 0.09%) + + + +doris::SegmentWriter::write_batch (28,202 samples, 18.19%) +doris::SegmentWriter::write_.. + + +doris::RunLengthIntegerWriter::_write_direct_values (49 samples, 0.03%) + + + +__GI___libc_write (38 samples, 0.02%) + + + +doris::CumulativeCompaction::_do_cumulative_compaction (67,697 samples, 43.67%) +doris::CumulativeCompaction::_do_cumulative_compaction + + +thread_group_cputime (25 samples, 0.02%) + + + +doris::IntegerColumnReaderWrapper<unsigned long, false>::next_vector (193 samples, 0.12%) + + + +doris::StorageByteBuffer::create (24 samples, 0.02%) + + + +doris::ser::bytes_to_long_be (71 samples, 0.05%) + + + +tcmalloc::CentralFreeList::InsertRange (28 samples, 0.02%) + + + +doris::ColumnDataWriter::mem_pool (21 samples, 0.01%) + + + +doris::FloatintPointColumnReader<double>::seek (59 samples, 0.04%) + + + +doris::OutStreamFactory::~OutStreamFactory (35 samples, 0.02%) + + + +doris::SegmentReader::get_block (40 samples, 0.03%) + + + +tcmalloc::CentralFreeList::FetchFromOneSpansSafe (17 samples, 0.01%) + + + +doris::ReadOnlyFileStream::_assure_data (103 samples, 0.07%) + + + +doris::StorageByteBuffer::put (51 samples, 0.03%) + + + +_raw_spin_lock (17 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_values (14 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_values (198 samples, 0.13%) + + + +doris::RowCursor::full_key_cmp (184 samples, 0.12%) + + + +doris::RowCursor::full_key_cmp (1,303 samples, 0.84%) + + + +doris::OutStream::write_to_file (47 samples, 0.03%) + + + +_raw_spin_lock (49 samples, 0.03%) + + + +doris::IntegerColumnWriterWrapper<long, true>::write_batch (47 samples, 0.03%) + + + +doris::FieldTypeTraits< (49 samples, 0.03%) + + + +doris::OLAPEngine::_cumulative_compaction_thread_callback (67,740 samples, 43.70%) +doris::OLAPEngine::_cumulative_compaction_thread_callback + + +clear_page_c_e (50 samples, 0.03%) + + + +tcmalloc::ThreadCache::Scavenge (26 samples, 0.02%) + + + +system_call (32 samples, 0.02%) + + + +doris::ser::read_ints (21 samples, 0.01%) + + + +tcmalloc::CentralFreeList::ReleaseToSpans (17 samples, 0.01%) + + + +inotify_inode_queue_event (14 samples, 0.01%) + + + +doris::RunLengthIntegerWriter::_determined_encoding (62 samples, 0.04%) + + + +doris::RunLengthIntegerReader::_read_delta_values (105 samples, 0.07%) + + + +doris::RowCursor::full_key_cmp (886 samples, 0.57%) + + + +doris::Conditions::append_condition (51 samples, 0.03%) + + + +doris::ColumnDataWriter::finalize (469 samples, 0.30%) + + + +doris::TaskWorkerPool::_report_olap_table_worker_thread_callback (153 samples, 0.10%) + + + +doris::ColumnData::_get_block (19 samples, 0.01%) + + + +doris::VarStringColumnWriter::write (27 samples, 0.02%) + + + +doris::StorageByteBuffer::put (40 samples, 0.03%) + + + +doris::ser::bytes_to_long_be (115 samples, 0.07%) + + + +intel_pmu_enable_all (49 samples, 0.03%) + + + +doris::VarStringColumnReader<doris::StringColumnDirectReader>::seek (159 samples, 0.10%) + + + +doris::ser::read_ints (527 samples, 0.34%) + + + +doris::RunLengthIntegerReader::_read_values (22 samples, 0.01%) + + + +doris::BaseCompaction::_do_base_compaction (23,342 samples, 15.06%) +doris::BaseCompaction::.. + + +tcmalloc::ThreadCache::Scavenge (28 samples, 0.02%) + + + +doris::RunLengthIntegerReader::_read_delta_values (39 samples, 0.03%) + + + +doris::FieldTypeTraits< (1,470 samples, 0.95%) + + + +radix_tree_lookup_slot (15 samples, 0.01%) + + + +doris::PInternalServiceImpl<doris::PBackendService>::_exec_plan_fragment (125 samples, 0.08%) + + + +doris::OutStream::_spill (75 samples, 0.05%) + + + +intel_pmu_enable_all (225 samples, 0.15%) + + + +doris::SegmentReader::_read_all_data_streams (52 samples, 0.03%) + + + +doris::RunLengthIntegerWriter::_determined_encoding (1,132 samples, 0.73%) + + + +doris::ser::read_var_unsigned (37 samples, 0.02%) + + + +doris::ser::read_ints (87 samples, 0.06%) + + + +doris::ColumnDataWriter::_finalize_segment (468 samples, 0.30%) + + + +doris::SegmentReader::init (16 samples, 0.01%) + + + +doris::StorageByteBuffer::put (18 samples, 0.01%) + + + +doris::ColumnStatistics::merge (69 samples, 0.04%) + + + +doris::PositionEntryReader::positions (35 samples, 0.02%) + + + +doris::ReadOnlyFileStream::_assure_data (24 samples, 0.02%) + + + +doris::IntegerColumnReaderWrapper<short, true>::next_vector (40 samples, 0.03%) + + + +__memcpy_sse2_unaligned (45 samples, 0.03%) + + + +doris::MemPool::free_all (37 samples, 0.02%) + + + +doris::RunLengthIntegerReader::_read_delta_values (797 samples, 0.51%) + + + +doris::ser::read_ints (364 samples, 0.23%) + + + +doris::StorageByteBuffer::put (26 samples, 0.02%) + + + +doris::ColumnData::prepare_block_read (2,052 samples, 1.32%) + + + +doris::ReadOnlyFileStream::_assure_data (40 samples, 0.03%) + + + +doris::olap_crc32 (48 samples, 0.03%) + + + +doris::RowCursor::full_key_cmp (3,695 samples, 2.38%) +d.. + + +doris::SegmentReader::_load_to_vectorized_row_batch (21,293 samples, 13.73%) +doris::SegmentReader.. + + +__memcpy_sse2_unaligned (19 samples, 0.01%) + + + +doris::FieldTypeTraits< (1,229 samples, 0.79%) + + + +clear_page_c_e (14 samples, 0.01%) + + + +doris::OutStream::write (27 samples, 0.02%) + + + +doris::RowCursor::full_key_cmp (272 samples, 0.18%) + + + +doris::FieldTypeTraits< (69 samples, 0.04%) + + + +syscall (80 samples, 0.05%) + + + +doris::RunLengthIntegerWriter::_write_values (261 samples, 0.17%) + + + +doris::IntegerColumnReaderWrapper<long, true>::seek (115 samples, 0.07%) + + + +doris::FieldTypeTraits< (270 samples, 0.17%) + + + +doris::ReadOnlyFileStream::_fill_compressed (21 samples, 0.01%) + + + +doris::RunLengthIntegerWriter::_write_short_repeat_values (221 samples, 0.14%) + + + +doris::StringColumnDirectReader::next_vector (58 samples, 0.04%) + + + +std::_Rb_tree<doris::StringValue, doris::StringValue, std::_Identity<doris::StringValue>, std::less<doris::StringValue>, std::allocator<doris::StringValue> >::find (85 samples, 0.05%) + + + +doris::RunLengthIntegerWriter::_write_short_repeat_values (72 samples, 0.05%) + + + +doris::FragmentExecState::prepare (83 samples, 0.05%) + + + +doris::RunLengthIntegerReader::_read_direct_values (136 samples, 0.09%) + + + +doris::OlapScanner::get_batch (2,175 samples, 1.40%) + + + +doris::OutStream::write (30 samples, 0.02%) + + + +__memcmp_sse4_1 (103 samples, 0.07%) + + + +doris::memory_copy (36 samples, 0.02%) + + + +doris::StorageByteBuffer::put (91 samples, 0.06%) + + + +doris::StorageByteBuffer::put (15 samples, 0.01%) + + + +_ZN5doris10ColumnData10_get_blockEbi.constprop.168 (40,153 samples, 25.90%) +_ZN5doris10ColumnData10_get_blockEbi.cons.. + + +doris::ReadOnlyFileStream::_assure_data (67 samples, 0.04%) + + + +doris::SegmentGroup::find_short_key (86 samples, 0.06%) + + + +doris::ReadOnlyFileStream::_assure_data (46 samples, 0.03%) + + + +tcmalloc::CentralFreeList::Populate (14 samples, 0.01%) + + + +thread_group_cputime (20 samples, 0.01%) + + + +doris::RunLengthIntegerWriter::_write_values (744 samples, 0.48%) + + + +__GI___strncmp_ssse3 (26 samples, 0.02%) + + + +doris::RunLengthIntegerReader::_read_values (46 samples, 0.03%) + + + +tcmalloc::CentralFreeList::RemoveRange (18 samples, 0.01%) + + + +thread_group_cputime (109 samples, 0.07%) + + + +brpc::Socket::ProcessEvent (156 samples, 0.10%) + + + +doris::RunLengthIntegerReader::_read_patched_base_values (822 samples, 0.53%) + + + +doris::RunLengthIntegerWriter::_determined_encoding (188 samples, 0.12%) + + + +doris::SegmentGroup::add_short_key (362 samples, 0.23%) + + + +doris::ColumnData::_seek_to_row (41,249 samples, 26.61%) +doris::ColumnData::_seek_to_row + + +doris::SegmentGroup::add_short_key (100 samples, 0.06%) + + + +doris::ReadOnlyFileStream::_assure_data (178 samples, 0.11%) + + + +doris::memory_copy (130 samples, 0.08%) + + + +doris::RunLengthIntegerReader::_read_values (1,583 samples, 1.02%) + + + +doris::RunLengthIntegerWriter::write (432 samples, 0.28%) + + + +doris::RunLengthIntegerReader::_read_values (90 samples, 0.06%) + + + +doris::ReadOnlyFileStream::_assure_data (59 samples, 0.04%) + + + +doris::ColumnData::_seek_to_row (2,052 samples, 1.32%) + + + +doris::RowCursor::allocate_memory_for_string_type (41 samples, 0.03%) + + + +__libc_send (36 samples, 0.02%) + + + +tcmalloc::CentralFreeList::FetchFromOneSpansSafe (26 samples, 0.02%) + + + +doris::FieldTypeTraits< (498 samples, 0.32%) + + + +doris::memory_copy (25 samples, 0.02%) + + + +doris::FieldTypeTraits< (181 samples, 0.12%) + + + +doris::IntegerColumnReaderWrapper<int, true>::next_vector (891 samples, 0.57%) + + + +tcmalloc::ThreadCache::ReleaseToCentralCache (37 samples, 0.02%) + + + +doris::memory_copy (38 samples, 0.02%) + + + +doris::RunLengthIntegerWriter::write (1,240 samples, 0.80%) + + + +doris::ReadOnlyFileStream::_assure_data (58 samples, 0.04%) + + + +find_busiest_group (56 samples, 0.04%) + + + +doris::RunLengthIntegerReader::_read_direct_values (529 samples, 0.34%) + + + +__audit_syscall_exit (61 samples, 0.04%) + + + +doris::StorageByteBuffer::create (15 samples, 0.01%) + + + +doris::Merger::merge (67,584 samples, 43.59%) +doris::Merger::merge + + +doris::ser::read_var_unsigned (40 samples, 0.03%) + + + +doris::ReadOnlyFileStream::_assure_data (34 samples, 0.02%) + + + +doris::TaskWorkerPool::_report_disk_state_worker_thread_callback (47 samples, 0.03%) + + + +doris::PBackendService::CallMethod (135 samples, 0.09%) + + + +doris::Reader::close (197 samples, 0.13%) + + + +doris::OutStream::_spill (17 samples, 0.01%) + + + +doris::OutStream::crc32 (82 samples, 0.05%) + + + +doris::StorageByteBuffer::put (169 samples, 0.11%) + + + +doris::IntegerColumnWriterWrapper<int, true>::write_batch (3,487 samples, 2.25%) +d.. + + +doris::ThreadPool::work_thread (75 samples, 0.05%) + + + +kmem_cache_alloc (16 samples, 0.01%) + + + +doris::RunLengthIntegerWriter::_write_delta_values (187 samples, 0.12%) + + + +doris::ReadOnlyFileStream::_assure_data (124 samples, 0.08%) + + + +doris::ReadOnlyFileStream::_assure_data (40 samples, 0.03%) + + + +release_pages (18 samples, 0.01%) + + + +doris::RunLengthIntegerWriter::_write_direct_values (527 samples, 0.34%) + + + +doris::ColumnWriter::finalize (27 samples, 0.02%) + + + +doris::ser::read_var_unsigned (15 samples, 0.01%) + + + +doris::RowCursor::_init (20 samples, 0.01%) + + + +doris::OutStream::write (385 samples, 0.25%) + + + +doris::RunLengthIntegerReader::_read_values (29 samples, 0.02%) + + + +doris::RunLengthIntegerWriter::write (13,860 samples, 8.94%) +doris::RunLe.. + + +doris::LzoBinaryReader::_next_block (45 samples, 0.03%) + + + +doris::ReadOnlyFileStream::_assure_data (28 samples, 0.02%) + + + +doris::FloatintPointColumnReader<double>::next_vector (283 samples, 0.18%) + + + +doris::OutStream::write (304 samples, 0.20%) + + + +get_page_from_freelist (14 samples, 0.01%) + + + +doris::PlanFragmentExecutor::open_internal (52 samples, 0.03%) + + + +doris::Reader::_attach_data_to_merge_set (2,053 samples, 1.32%) + + + +doris::RunLengthIntegerReader::_read_delta_values (168 samples, 0.11%) + + + +doris::OutStream::write (160 samples, 0.10%) + + + +doris::Reader::_init_conditions_param (67 samples, 0.04%) + + + +doris::FieldTypeTraits< (383 samples, 0.25%) + + + +doris::RowCursor::full_key_cmp (586 samples, 0.38%) + + + +doris::FragmentMgr::exec_actual (61 samples, 0.04%) + + + +palo_be (155,028 samples, 100.00%) +palo_be + + +LZ4_decompress_safe (8,365 samples, 5.40%) +LZ4_de.. + + +doris::OlapScanner::open (41,543 samples, 26.80%) +doris::OlapScanner::open + + +doris::memory_copy (20 samples, 0.01%) + + + +thread_group_cputime (19 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_values (295 samples, 0.19%) + + + +put_page (44 samples, 0.03%) + + + +LZ4_compress_fast_extState (2,610 samples, 1.68%) + + + +doris::ser::write_ints (908 samples, 0.59%) + + + +doris::memory_copy (170 samples, 0.11%) + + + +doris::LargeIntColumnReader::next_vector (15 samples, 0.01%) + + + +doris::ReadOnlyFileStream::_assure_data (26 samples, 0.02%) + + + +doris::Reader::_init_params (106 samples, 0.07%) + + + +doris::LargeIntColumnWriter::write_batch (19,822 samples, 12.79%) +doris::LargeIntColu.. + + +doris::RunLengthIntegerReader::_read_values (2,131 samples, 1.37%) + + + +doris::ser::write_ints (39 samples, 0.03%) + + + +doris::EqualPredicate<doris::StringValue>::evaluate (498 samples, 0.32%) + + + +doris::RowCursor::allocate_memory_for_string_type (843 samples, 0.54%) + + + +doris::StorageByteBuffer::put (15 samples, 0.01%) + + + +doris::ser::write_var_unsigned (25 samples, 0.02%) + + + +doris::ReadOnlyFileStream::seek (21 samples, 0.01%) + + + +doris::MemIndex::find (77 samples, 0.05%) + + + +doris::OLAPTable::get_num_rows (23 samples, 0.01%) + + + +flush_tlb_others_ipi (43 samples, 0.03%) + + + +doris::RunLengthIntegerWriter::_determined_encoding (127 samples, 0.08%) + + + +doris::ser::read_var_unsigned (218 samples, 0.14%) + + + +tcmalloc::ThreadCache::FetchFromCentralCache (29 samples, 0.02%) + + + +doris::ser::read_ints (90 samples, 0.06%) + + + +thread_group_cputime (24 samples, 0.02%) + + + +tcmalloc::CentralFreeList::ReleaseListToSpans (16 samples, 0.01%) + + + +doris::PositionEntryReader::positions (19 samples, 0.01%) + + + +all (155,028 samples, 100%) + + + +mnt_want_write_file (14 samples, 0.01%) + + + +thread_group_cputime (16 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_direct_values (205 samples, 0.13%) + + + +doris::IntegerColumnReaderWrapper<long, true>::init (25 samples, 0.02%) + + + +ext4_da_get_block_prep (14 samples, 0.01%) + + + +doris::OutStream::write (28 samples, 0.02%) + + + +free_pcppages_bulk (18 samples, 0.01%) + + + +__mem_cgroup_uncharge_common (29 samples, 0.02%) + + + +doris::MemPool::allocate (56 samples, 0.04%) + + + +tcmalloc::CentralFreeList::RemoveRange (35 samples, 0.02%) + + + +tcmalloc::CentralFreeList::Populate (52 samples, 0.03%) + + + +__GI___libc_write (88 samples, 0.06%) + + + +doris::RunLengthIntegerReader::_read_direct_values (41 samples, 0.03%) + + + +doris::RunLengthIntegerReader::seek (17 samples, 0.01%) + + + +doris::FieldTypeTraits< (91 samples, 0.06%) + + + + (6,027 samples, 3.89%) + + + +pthread_cond_wait@@GLIBC_2.3.2 (280 samples, 0.18%) + + + +doris::MemPool::FindChunk (20 samples, 0.01%) + + + +doris::ser::write_ints (111 samples, 0.07%) + + + +doris::RunLengthIntegerReader::_read_direct_values (31 samples, 0.02%) + + + +_raw_spin_lock (15 samples, 0.01%) + + + +touch_atime (54 samples, 0.03%) + + + +generic_file_aio_read (15 samples, 0.01%) + + + +tcmalloc::CentralFreeList::ReleaseListToSpans (18 samples, 0.01%) + + + +doris::StorageByteBuffer::put (204 samples, 0.13%) + + + +doris::OLAPEngine::perform_base_compaction (23,347 samples, 15.06%) +doris::OLAPEngine::perf.. + + +tcmalloc::ThreadCache::Scavenge (38 samples, 0.02%) + + + +doris::RunLengthIntegerReader::_read_direct_values (21 samples, 0.01%) + + + +doris::VarStringColumnWriter::write (587 samples, 0.38%) + + + +do_sync_read (21 samples, 0.01%) + + + +doris::CumulativeCompaction::run (67,702 samples, 43.67%) +doris::CumulativeCompaction::run + + +doris::StorageByteBuffer::put (167 samples, 0.11%) + + + +doris::IntegerColumnReaderWrapper<short, true>::next_vector (3,582 samples, 2.31%) +d.. + + +doris::RowCursor::allocate_memory_for_string_type (91 samples, 0.06%) + + + +doris::StringColumnDirectReader::next_vector (349 samples, 0.23%) + + + +doris::RunLengthIntegerWriter::_write_values (409 samples, 0.26%) + + + +doris::SegmentWriter::finalize (433 samples, 0.28%) + + + +doris::SegmentWriter::write_batch (74 samples, 0.05%) + + + +doris::RunLengthIntegerReader::_read_values (113 samples, 0.07%) + + + +doris::TExecPlanFragmentParams::read (23 samples, 0.01%) + + + +doris::ser::read_ints (592 samples, 0.38%) + + + +thread_group_cputime (16 samples, 0.01%) + + + +std::_Rb_tree<doris::StringValue, doris::StringValue, std::_Identity<doris::StringValue>, std::less<doris::StringValue>, std::allocator<doris::StringValue> >::find (259 samples, 0.17%) + + + +doris::RunLengthIntegerWriter::_determined_encoding (592 samples, 0.38%) + + + +tcmalloc::ThreadCache::ReleaseToCentralCache (26 samples, 0.02%) + + + +doris::PriorityThreadPool::work_thread (43,966 samples, 28.36%) +doris::PriorityThreadPool::work_thread + + +doris::ser::write_ints (38 samples, 0.02%) + + + +doris::ColumnDataWriter::attached_by (31,819 samples, 20.52%) +doris::ColumnDataWriter::attache.. + + +doris::OutStream::write (19 samples, 0.01%) + + + +thread_group_cputime (99 samples, 0.06%) + + + +doris::ReadOnlyFileStream::_assure_data (26 samples, 0.02%) + + + +tcmalloc::CentralFreeList::Populate (21 samples, 0.01%) + + + +doris::ser::read_ints (25 samples, 0.02%) + + + +doris::RunLengthIntegerWriter::_determined_encoding (597 samples, 0.39%) + + + +doris::InListPredicate<doris::StringValue>::evaluate (15,540 samples, 10.02%) +doris::InListP.. + + +doris::RunLengthIntegerReader::seek (19 samples, 0.01%) + + + +tcmalloc::ThreadCache::FetchFromCentralCache (21 samples, 0.01%) + + + +thread_group_cputime (105 samples, 0.07%) + + + + (236 samples, 0.15%) + + + +doris::StringColumnDirectReader::next_vector (306 samples, 0.20%) + + + +doris::FrontendServiceClient::send_report (34 samples, 0.02%) + + + +doris::ReadOnlyFileStream::seek (75 samples, 0.05%) + + + +doris::RunLengthIntegerReader::_read_values (37 samples, 0.02%) + + + +doris::RunLengthIntegerWriter::write (904 samples, 0.58%) + + + +doris::ColumnData::_find_position_by_short_key (94 samples, 0.06%) + + + +doris::OutStream::~OutStream (34 samples, 0.02%) + + + +doris::ReadOnlyFileStream::seek (44 samples, 0.03%) + + + +std::pair<std::__detail::_Node_iterator<doris::WrapperField const*, true, true>, bool> std::_Hashtable<doris::WrapperField const*, doris::WrapperField const*, std::allocator<doris::WrapperField const*>, std::__detail::_Identity, doris::FieldEqual, doris::FieldHash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::_M_insert<doris::WrapperField const*, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<doris::WrapperField const*, true> > > > (25 samples, 0.02%) + + + +std::pair<std::_Rb_tree_iterator<std::pair<long const, doris::TTablet> >, bool> std::_Rb_tree<long, std::pair<long const, doris::TTablet>, std::_Select1st<std::pair<long const, doris::TTablet> >, std::less<long>, std::allocator<std::pair<long const, doris::TTablet> > >::_M_insert_unique<std::pair<long, doris::TTablet> > (22 samples, 0.01%) + + + +doris::VectorizedRowBatch::dump_to_row_block (1,512 samples, 0.98%) + + + +LZ4_decompress_safe (20 samples, 0.01%) + + + +doris::LargeIntColumnWriter::finalize (29 samples, 0.02%) + + + +doris::RunLengthIntegerWriter::_write_values (2,810 samples, 1.81%) +d.. + + +doris::ser::bytes_to_long_be (72 samples, 0.05%) + + + +doris::ser::bytes_to_long_be (158 samples, 0.10%) + + + +doris::StorageByteBuffer::put (20 samples, 0.01%) + + + +doris::ExecNode::create_tree_helper (32 samples, 0.02%) + + + +doris::ReadOnlyFileStream::_assure_data (22 samples, 0.01%) + + + +doris::NewPartitionedAggregationNode::get_next (15 samples, 0.01%) + + + +doris::ser::read_var_unsigned (231 samples, 0.15%) + + + +tcmalloc::ThreadCache::ReleaseToCentralCache (27 samples, 0.02%) + + + +page_remove_rmap (14 samples, 0.01%) + + + +std::_Rb_tree<doris::StringValue, doris::StringValue, std::_Identity<doris::StringValue>, std::less<doris::StringValue>, std::allocator<doris::StringValue> >::find (13,926 samples, 8.98%) +std::_Rb_tre.. + + +doris::RunLengthIntegerReader::_read_values (84 samples, 0.05%) + + + +doris::OLAPTable::release_data_sources (185 samples, 0.12%) + + + +doris::ColumnWriter::create_row_index_entry (63 samples, 0.04%) + + + +system_call_after_swapgs (60 samples, 0.04%) + + + +doris::ReadOnlyFileStream::_assure_data (57 samples, 0.04%) + + + +doris::ser::write_ints (476 samples, 0.31%) + + + +doris::RunLengthIntegerWriter::_write_delta_values (22 samples, 0.01%) + + + +doris::RunLengthIntegerWriter::_write_direct_values (1,028 samples, 0.66%) + + + +doris::ColumnData::get_next_block (10,553 samples, 6.81%) +doris::Co.. + + +doris::SegmentReader::get_block (22,235 samples, 14.34%) +doris::SegmentReader:.. + + +doris::IntegerColumnReaderWrapper<int, true>::next_vector (2,822 samples, 1.82%) +d.. + + +doris::FloatintPointColumnReader<double>::next_vector (155 samples, 0.10%) + + + +doris::RowCursor::cmp (217 samples, 0.14%) + + + +doris::StorageByteBuffer::create (14 samples, 0.01%) + + + +doris::VectorizedRowBatch::dump_to_row_block (1,870 samples, 1.21%) + + + +__memcmp_sse4_1 (14 samples, 0.01%) + + + +doris::MemPool::clear (17 samples, 0.01%) + + + +tcmalloc::ThreadCache::FetchFromCentralCache (35 samples, 0.02%) + + + +doris::SegmentWriter::~SegmentWriter (34 samples, 0.02%) + + + +tcmalloc::CentralFreeList::Populate (18 samples, 0.01%) + + + +doris::IntegerColumnWriterWrapper<long, true>::write_batch (3,196 samples, 2.06%) +d.. + + +doris::RunLengthIntegerWriter::_write_short_repeat_values (72 samples, 0.05%) + + + +doris::OutStream::write (51 samples, 0.03%) + + + +doris::OLAPEngine::_base_compaction_thread_callback (23,349 samples, 15.06%) +doris::OLAPEngine::_bas.. + + +doris::FieldTypeTraits< (746 samples, 0.48%) + + + +copy_user_generic_string (92 samples, 0.06%) + + + +thread_group_cputime (20 samples, 0.01%) + + + +doris::LargeIntColumnReader::init (36 samples, 0.02%) + + + +thread_group_cputime (138 samples, 0.09%) + + + + (185 samples, 0.12%) + + + +doris::IntegerColumnWriterWrapper<short, true>::write_batch (22 samples, 0.01%) + + + +doris::Reader::~Reader (212 samples, 0.14%) + + + +copy_user_generic_string (117 samples, 0.08%) + + + +doris::VectorizedRowBatch::VectorizedRowBatch (19 samples, 0.01%) + + + +doris::ser::write_ints (43 samples, 0.03%) + + + +doris::OLAPTable::get_data_size (27 samples, 0.02%) + + + +doris::ser::write_ints (879 samples, 0.57%) + + + +pthread_cond_wait@@GLIBC_2.3.2 (364 samples, 0.23%) + + + +doris::StorageByteBuffer::create (36 samples, 0.02%) + + + +clear_page_c_e (16 samples, 0.01%) + + + +std::_Rb_tree<doris::StringValue, doris::StringValue, std::_Identity<doris::StringValue>, std::less<doris::StringValue>, std::allocator<doris::StringValue> >::find (856 samples, 0.55%) + + + +doris::OutStream::write (25 samples, 0.02%) + + + +_ZN5doris10ColumnData10_get_blockEbi.constprop.168 (68 samples, 0.04%) + + + +lzo1x_decompress_safe (45 samples, 0.03%) + + + +bvar::detail::SamplerCollector::run (61 samples, 0.04%) + + + +tcmalloc::CentralFreeList::Populate (17 samples, 0.01%) + + + +doris::OLAPEngine::report_all_tablets_info (104 samples, 0.07%) + + + +doris::ser::read_ints (432 samples, 0.28%) + + + +doris::StorageByteBuffer::put (19 samples, 0.01%) + + + +doris::memory_copy (147 samples, 0.09%) + + + +tcmalloc::CentralFreeList::FetchFromOneSpansSafe (21 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_delta_values (49 samples, 0.03%) + + + +__GI___libc_nanosleep (89 samples, 0.06%) + + + +doris::ser::read_ints (168 samples, 0.11%) + + + +doris::ReadOnlyFileStream::_assure_data (22 samples, 0.01%) + + + +doris::TPlanFragment::read (20 samples, 0.01%) + + + +doris::ExecNode::eval_conjuncts (19 samples, 0.01%) + + + +doris::ColumnDataWriter::attached_by (77 samples, 0.05%) + + + +tcmalloc::CentralFreeList::InsertRange (22 samples, 0.01%) + + + +doris::SegmentWriter::estimate_segment_size (1,790 samples, 1.15%) + + + +doris::StreamIndexReader::init (18 samples, 0.01%) + + + +get_pageblock_flags_group (14 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_patched_base_values (14 samples, 0.01%) + + + +doris::ser::find_closet_num_bits (15 samples, 0.01%) + + + +doris::ser::read_ints (145 samples, 0.09%) + + + +doris::OutStream::write (63 samples, 0.04%) + + + +__memcmp_sse4_1 (114 samples, 0.07%) + + + +doris::TTabletInfo::write (21 samples, 0.01%) + + + +doris::RowCursor::_init (17 samples, 0.01%) + + + +doris::TReportRequest::write (25 samples, 0.02%) + + + +doris::RunLengthIntegerReader::_read_values (18 samples, 0.01%) + + + +thread_group_cputime (47 samples, 0.03%) + + + +doris::FieldTypeTraits< (1,818 samples, 1.17%) + + + +doris::RunLengthIntegerWriter::_write_patched_base_values (1,058 samples, 0.68%) + + + +doris::RunLengthIntegerReader::_read_values (1,947 samples, 1.26%) + + + +doris::RunLengthIntegerWriter::write (195 samples, 0.13%) + + + +bthread::TaskGroup::task_runner (169 samples, 0.11%) + + + +doris::Reader::_agg_key_next_row (2,135 samples, 1.38%) + + + +doris::LargeIntColumnReader::seek (19 samples, 0.01%) + + + +doris::memory_copy (17 samples, 0.01%) + + + +doris::VectorizedRowBatch::dump_to_row_block (4,670 samples, 3.01%) +dor.. + + +doris::RunLengthIntegerWriter::write (2,016 samples, 1.30%) + + + +doris::StreamIndexWriter::add_index_entry (25 samples, 0.02%) + + + +doris::OlapScanNode::scanner_thread (43,963 samples, 28.36%) +doris::OlapScanNode::scanner_thread + + +thread_group_cputime (61 samples, 0.04%) + + + +doris::ReadOnlyFileStream::_assure_data (1,232 samples, 0.79%) + + + +_ZN5doris10ColumnData10_get_blockEbi.constprop.168 (14 samples, 0.01%) + + + +doris::RunLengthIntegerWriter::_write_values (23 samples, 0.01%) + + + +__might_sleep (20 samples, 0.01%) + + + +doris::PlanFragmentExecutor::get_next_internal (16 samples, 0.01%) + + + +bvar::detail::SamplerCollector::sampling_thread (67 samples, 0.04%) + + + +doris::OlapScanner::close (217 samples, 0.14%) + + + + (91,100 samples, 58.76%) + + + +doris::ReadOnlyFileStream::_assure_data (685 samples, 0.44%) + + + +doris::RunLengthIntegerReader::_read_values (2,546 samples, 1.64%) + + + +doris::OutStream::write (33 samples, 0.02%) + + + +memset (21 samples, 0.01%) + + + +doris::RowBlock::init (34 samples, 0.02%) + + + +doris::SegmentWriter::write_batch (7,502 samples, 4.84%) +doris:.. + + +doris::RunLengthIntegerReader::_read_values (59 samples, 0.04%) + + + +doris::ser::read_var_unsigned (156 samples, 0.10%) + + + +posix_fadvise64 (280 samples, 0.18%) + + + +doris::SegmentReader::_load_index (61 samples, 0.04%) + + + +doris::RunLengthIntegerWriter::_write_delta_values (86 samples, 0.06%) + + + +doris::OutStream::_spill (29 samples, 0.02%) + + + +doris::ReadOnlyFileStream::_assure_data (168 samples, 0.11%) + + + +doris::SegmentReader::init (110 samples, 0.07%) + + + +doris::RunLengthIntegerReader::_read_direct_values (686 samples, 0.44%) + + + +clear_page_c_e (26 samples, 0.02%) + + + +doris::OLAPEngine::perform_cumulative_compaction (67,733 samples, 43.69%) +doris::OLAPEngine::perform_cumulative_compaction + + +doris::RunLengthIntegerReader::_read_direct_values (26 samples, 0.02%) + + + +tcmalloc::CentralFreeList::Populate (26 samples, 0.02%) + + + +doris::RunLengthIntegerReader::_read_values (1,416 samples, 0.91%) + + + +doris::MemPool::free_all (15 samples, 0.01%) + + + +doris::Reader::_agg_key_next_row (147 samples, 0.09%) + + + +doris::OLAPTable::get_missing_versions_with_header_locked (19 samples, 0.01%) + + + +tcmalloc::ThreadCache::Scavenge (19 samples, 0.01%) + + + +doris::FieldTypeTraits< (98 samples, 0.06%) + + + +doris::PlanFragmentExecutor::prepare (83 samples, 0.05%) + + + +doris::ReadOnlyFileStream::_assure_data (31 samples, 0.02%) + + + +tcmalloc::CentralFreeList::RemoveRange (21 samples, 0.01%) + + + +vfs_read (52 samples, 0.03%) + + + +__GI___libc_write (914 samples, 0.59%) + + + +doris::Reader::_agg_key_next_row (30 samples, 0.02%) + + + +tcmalloc::CentralFreeList::RemoveRange (21 samples, 0.01%) + + + +doris::VarStringColumnWriter::write_batch (702 samples, 0.45%) + + + +doris::OLAPEngine::get_all_root_path_info (36 samples, 0.02%) + + + +doris::ExecNode::create_tree_helper (15 samples, 0.01%) + + + +doris::ReadOnlyFileStream::_assure_data (67 samples, 0.04%) + + + +doris::OutStream::write (35 samples, 0.02%) + + + +doris::memory_copy (328 samples, 0.21%) + + + +__clone (17 samples, 0.01%) + + + +doris::RunLengthIntegerWriter::_write_delta_values (336 samples, 0.22%) + + + +bvar::PassiveStatus<long>::SeriesSampler::take_sample (22 samples, 0.01%) + + + +pthread_cond_timedwait@@GLIBC_2.3.2 (15 samples, 0.01%) + + + +generic_file_aio_read (162 samples, 0.10%) + + + +doris::ser::write_ints (45 samples, 0.03%) + + + +doris::FloatintPointColumnReader<double>::next_vector (44 samples, 0.03%) + + + +doris::NewPartitionedAggregationNode::prepare (23 samples, 0.01%) + + + +doris::memory_copy (329 samples, 0.21%) + + + +std::_Rb_tree_increment (26 samples, 0.02%) + + + +doris::ser::write_var_unsigned (67 samples, 0.04%) + + + +copy_user_generic_string (14 samples, 0.01%) + + + +doris::ser::bytes_to_long_be (26 samples, 0.02%) + + + +doris::RunLengthIntegerReader::_read_delta_values (143 samples, 0.09%) + + + +doris::TPlan::read (19 samples, 0.01%) + + + +doris::Merger::merge (23,317 samples, 15.04%) +doris::Merger::merge + + +doris::ColumnData::get_next_block (3,654 samples, 2.36%) +d.. + + +_ZN5doris10ColumnData10_get_blockEbi.constprop.168 (3,645 samples, 2.35%) +_.. + + +doris::RunLengthIntegerReader::_read_delta_values (17 samples, 0.01%) + + + +strncmp@plt (33 samples, 0.02%) + + + +doris::IntegerColumnReaderWrapper<long, true>::next_vector (24 samples, 0.02%) + + + +doris::memory_copy (959 samples, 0.62%) + + + +doris::ColumnData::_seek_to_block (548 samples, 0.35%) + + + +doris::RunLengthIntegerReader::_read_delta_values (650 samples, 0.42%) + + + +doris::ReadOnlyFileStream::_fill_compressed (23 samples, 0.01%) + + + +doris::memory_copy (77 samples, 0.05%) + + + +doris::ColumnDataWriter::next (2,099 samples, 1.35%) + + + +doris::memory_copy (232 samples, 0.15%) + + + +bvar::detail::ReducerSampler<bvar::detail::Percentile, bvar::detail::PercentileSamples<254ul>, bvar::detail::Percentile::AddPercentileSamples, bvar::detail::VoidOp>::take_sample (18 samples, 0.01%) + + + +doris::ReadOnlyFileStream::seek (26 samples, 0.02%) + + + +doris::ser::find_closet_num_bits (15 samples, 0.01%) + + + +doris::StringColumnDirectReader::next_vector (8,135 samples, 5.25%) +doris:.. + + +doris::FieldTypeTraits< (116 samples, 0.07%) + + + +__rmqueue (14 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_direct_values (193 samples, 0.12%) + + + +doris::ReadOnlyFileStream::_assure_data (18 samples, 0.01%) + + + +doris::VarStringColumnReader<doris::StringColumnDirectReader>::next_vector (8,523 samples, 5.50%) +doris::.. + + +tcmalloc::ThreadCache::FetchFromCentralCache (14 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_delta_values (15 samples, 0.01%) + + + +_ZN5doris10ColumnData10_get_blockEbi.constprop.168 (2,032 samples, 1.31%) + + + +doris::OutStreamFactory::~OutStreamFactory (34 samples, 0.02%) + + + +bvar::detail::AgentCombiner<bvar::detail::PercentileSamples<254ul>, bvar::detail::PercentileSamples<30ul>, bvar::detail::Percentile::AddPercentileSamples>::reset_all_agents (15 samples, 0.01%) + + + +doris::OutStream::write_to_file (109 samples, 0.07%) + + + +doris::PlanFragmentExecutor::open (54 samples, 0.03%) + + + +doris::StorageByteBuffer::create (31 samples, 0.02%) + + + +tcmalloc::ThreadCache::FetchFromCentralCache (59 samples, 0.04%) + + + +doris::Reader::init (41,542 samples, 26.80%) +doris::Reader::init + + +doris::IntegerColumnReaderWrapper<long, true>::next_vector (69 samples, 0.04%) + + + +doris::NewPartitionedAggregationNode::GetRowsStreaming (14 samples, 0.01%) + + + +doris::ColumnData::get_next_block (70 samples, 0.05%) + + + +doris::StorageByteBuffer::put (29 samples, 0.02%) + + + +doris::ColumnDataWriter::next (105 samples, 0.07%) + + + +doris::ColumnDataWriter::next (41 samples, 0.03%) + + + +futex_wait (14 samples, 0.01%) + + + +fget_light (15 samples, 0.01%) + + + +tcmalloc::ThreadCache::FetchFromCentralCache (18 samples, 0.01%) + + + +doris::TTablet::write (25 samples, 0.02%) + + + +doris::ser::bytes_to_long_be (52 samples, 0.03%) + + + +doris::FieldTypeTraits< (54 samples, 0.03%) + + + +doris::RunLengthIntegerReader::_read_values (606 samples, 0.39%) + + + +doris::RowCursor::_init (21 samples, 0.01%) + + + +doris::ColumnWriter::create_row_index_entry (175 samples, 0.11%) + + + +find_get_page (203 samples, 0.13%) + + + +doris::RunLengthIntegerReader::_read_direct_values (456 samples, 0.29%) + + + +doris::OutStream::~OutStream (35 samples, 0.02%) + + + +brpc::InputMessenger::OnNewMessages (155 samples, 0.10%) + + + +thread_group_cputime (69 samples, 0.04%) + + + +generic_file_aio_read (17 samples, 0.01%) + + + +doris::StorageByteBuffer::put (104 samples, 0.07%) + + + +smp_invalidate_interrupt (23 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_delta_values (26 samples, 0.02%) + + + +doris::OutStream::_spill (18 samples, 0.01%) + + + +doris::ReadOnlyFileStream::seek (18 samples, 0.01%) + + + +doris::FloatintPointColumnReader<double>::next_vector (2,723 samples, 1.76%) + + + +std::_Rb_tree_increment (36 samples, 0.02%) + + + +doris::NewPartitionedAggregationNode::GetNextInternal (15 samples, 0.01%) + + + +doris::OLAPHeader::select_versions_to_span (19 samples, 0.01%) + + + +_ZN5doris10ColumnData10_get_blockEbi.constprop.168 (10,521 samples, 6.79%) +_ZN5doris.. + + +doris::Reader::_init_keys_param (29 samples, 0.02%) + + + +doris::SegmentReader::_set_segment_info (22 samples, 0.01%) + + + +doris::ser::read_var_unsigned (118 samples, 0.08%) + + + + (118 samples, 0.08%) + + + +doris::WrapperField::create_by_type (14 samples, 0.01%) + + + +fsnotify (25 samples, 0.02%) + + + +doris::ColumnData::_find_position_by_full_key (37 samples, 0.02%) + + + +doris::ReadOnlyFileStream::_assure_data (41 samples, 0.03%) + + + +__GI___libc_nanosleep (184 samples, 0.12%) + + + +doris::memory_copy (111 samples, 0.07%) + + + +doris::ser::bytes_to_long_be (32 samples, 0.02%) + + + +doris::VarStringColumnWriter::write_batch (39 samples, 0.03%) + + + +mem_cgroup_del_lru_list (16 samples, 0.01%) + + + +doris::SegmentReader::get_block (1,000 samples, 0.65%) + + + +thread_group_cputime (20 samples, 0.01%) + + + +tcmalloc::CentralFreeList::FetchFromOneSpansSafe (19 samples, 0.01%) + + + +thread_group_cputime (25 samples, 0.02%) + + + +doris::Cond::init (48 samples, 0.03%) + + + +doris::StorageByteBuffer::put (56 samples, 0.04%) + + + +__mem_cgroup_uncharge_common (21 samples, 0.01%) + + + +__block_prepare_write (30 samples, 0.02%) + + + +doris::SegmentReader::~SegmentReader (119 samples, 0.08%) + + + +doris::RowCursor::init_scan_key (25 samples, 0.02%) + + + +doris::ReadOnlyFileStream::_fill_compressed (18 samples, 0.01%) + + + +doris::RunLengthIntegerWriter::write (138 samples, 0.09%) + + + +fget_light (101 samples, 0.07%) + + + +doris::SegmentReader::_load_to_vectorized_row_batch (947 samples, 0.61%) + + + +tcmalloc::CentralFreeList::FetchFromOneSpansSafe (31 samples, 0.02%) + + + +doris::StorageByteBuffer::put (19 samples, 0.01%) + + + +doris::RunLengthIntegerReader::seek (58 samples, 0.04%) + + + +finish (169 samples, 0.11%) + + + +doris::VectorizedRowBatch::dump_to_row_block (110 samples, 0.07%) + + + +doris::RunLengthIntegerReader::seek (67 samples, 0.04%) + + + +vfs_write (14 samples, 0.01%) + + + +tcmalloc::CentralFreeList::FetchFromOneSpansSafe (14 samples, 0.01%) + + + +doris::ColumnDataWriter::next (3,662 samples, 2.36%) +d.. + + +doris::ser::read_ints (35 samples, 0.02%) + + + +execute_native_thread_routine (91,100 samples, 58.76%) +execute_native_thread_routine + + +doris::FragmentMgr::exec_plan_fragment (84 samples, 0.05%) + + + +unmap_vmas (31 samples, 0.02%) + + + +doris::RowCursor::full_key_cmp (3,630 samples, 2.34%) +d.. + + +_ZNSt6vectorIN5doris19PositionEntryWriterESaIS1_EE17_M_realloc_insertIJRKS1_EEEvN9__gnu_cxx17__normal_iteratorIPS1_S3_EEDpOT_ (43 samples, 0.03%) + + + +kmem_cache_alloc (21 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_values (24 samples, 0.02%) + + + +doris::Reader::_agg_key_next_row (26,822 samples, 17.30%) +doris::Reader::_agg_key_ne.. + + +doris::OutStream::get_total_buffer_size (47 samples, 0.03%) + + + +doris::IntegerColumnReaderWrapper<long, true>::next_vector (700 samples, 0.45%) + + + +doris::ReadOnlyFileStream::_assure_data (14 samples, 0.01%) + + + +tcmalloc::CentralFreeList::RemoveRange (29 samples, 0.02%) + + + +doris::RunLengthIntegerReader::_read_delta_values (17 samples, 0.01%) + + + +doris::RunLengthIntegerReader::seek (110 samples, 0.07%) + + + +doris::OutStream::crc32 (52 samples, 0.03%) + + + +doris::TTablet::TTablet (17 samples, 0.01%) + + + +doris::ser::bytes_to_long_be (522 samples, 0.34%) + + + +doris::TaskWorkerPool::_push_worker_thread_callback (219 samples, 0.14%) + + + +doris::FrontendServiceClient::report (34 samples, 0.02%) + + + +doris::SegmentReader::_load_to_vectorized_row_batch (40 samples, 0.03%) + + + +doris::IntegerColumnWriterWrapper<short, true>::write_batch (1,772 samples, 1.14%) + + + +__memcpy_sse2_unaligned (48 samples, 0.03%) + + + +doris::BinarySearchIterator std::__lower_bound<doris::BinarySearchIterator, doris::RowCursor, __gnu_cxx::__ops::_Iter_comp_val<doris::ColumnDataComparator> > (35 samples, 0.02%) + + + + (17 samples, 0.01%) + + + +doris::memory_copy (1,156 samples, 0.75%) + + + +doris::RunLengthIntegerReader::seek (297 samples, 0.19%) + + + +doris::SegmentReader::_load_to_vectorized_row_batch (5,484 samples, 3.54%) +dor.. + + +doris::RunLengthIntegerWriter::_determined_encoding (6,559 samples, 4.23%) +doris.. + + +find_get_page (26 samples, 0.02%) + + + +file_read_actor (50 samples, 0.03%) + + + +void std::__adjust_heap<__gnu_cxx::__normal_iterator<doris::CollectIterator::ChildCtx**, std::vector<doris::CollectIterator::ChildCtx*, std::allocator<doris::CollectIterator::ChildCtx*> > >, long, doris::CollectIterator::ChildCtx*, __gnu_cxx::__ops::_Iter_comp_iter<doris::CollectIterator::ChildCtxComparator> > (48 samples, 0.03%) + + + +tcmalloc::CentralFreeList::Populate (29 samples, 0.02%) + + + +doris::ColumnStatistics::init (14 samples, 0.01%) + + + +doris::LargeIntColumnReader::next_vector (169 samples, 0.11%) + + + +doris::RunLengthIntegerReader::_read_delta_values (418 samples, 0.27%) + + + +doris::SegmentReader::get_block (5,506 samples, 3.55%) +dor.. + + +doris::RunLengthIntegerReader::_read_delta_values (515 samples, 0.33%) + + + +tcmalloc::ThreadCache::FetchFromCentralCache (21 samples, 0.01%) + + + +thread_group_cputime (18 samples, 0.01%) + + + +doris::RunLengthIntegerWriter::write (146 samples, 0.09%) + + + +doris::OutStream::write (17 samples, 0.01%) + + + +memset (833 samples, 0.54%) + + + +doris::Status doris::deserialize_thrift_msg<doris::TExecPlanFragmentParams> (32 samples, 0.02%) + + + +doris::ColumnStatistics::attach (62 samples, 0.04%) + + + +doris::ReadOnlyFileStream::_assure_data (16 samples, 0.01%) + + + +doris::FieldTypeTraits< (58 samples, 0.04%) + + + +tcmalloc::CentralFreeList::ReleaseToSpans (25 samples, 0.02%) + + + +thread_group_cputime (28 samples, 0.02%) + + + +doris::Pusher::process (216 samples, 0.14%) + + + +doris::ReadOnlyFileStream::_assure_data (53 samples, 0.03%) + + + +doris::IntegerColumnReaderWrapper<long, true>::next_vector (544 samples, 0.35%) + + + +doris::LzoBinaryReader::next (77 samples, 0.05%) + + + +boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, doris::FragmentMgr, std::shared_ptr<doris::FragmentExecState>, std::function<void (61 samples, 0.04%) + + + +doris::RunLengthIntegerWriter::_determined_encoding (18 samples, 0.01%) + + + +doris::OutStream::get_total_buffer_size (94 samples, 0.06%) + + + +doris::BaseCompaction::run (23,342 samples, 15.06%) +doris::BaseCompaction::.. + + +doris::ReadOnlyFileStream::_assure_data (103 samples, 0.07%) + + + +thread_group_cputime (43 samples, 0.03%) + + + +doris::OLAPTable::acquire_data_sources_by_versions (33 samples, 0.02%) + + + +doris::PushHandler::process_realtime_push (215 samples, 0.14%) + + + +doris::StreamIndexWriter::add_index_entry (43 samples, 0.03%) + + + +doris::ReadOnlyFileStream::_assure_data (27 samples, 0.02%) + + + +doris::WrapperField::create (18 samples, 0.01%) + + + +doris::ReadOnlyFileStream::_fill_compressed (213 samples, 0.14%) + + + +doris::StreamIndexReader::entry (42 samples, 0.03%) + + + +doris::RunLengthIntegerReader::_read_values (409 samples, 0.26%) + + + +doris::FrontendService_report_pargs::write (34 samples, 0.02%) + + + +LZ4_compress_fast_extState (53 samples, 0.03%) + + + +doris::RunLengthIntegerReader::_read_delta_values (20 samples, 0.01%) + + + +fput (22 samples, 0.01%) + + + +doris::IntegerColumnWriterWrapper<int, true>::write_batch (40 samples, 0.03%) + + + +doris::CondColumn::eval (70 samples, 0.05%) + + + +doris::IntegerColumnReaderWrapper<int, true>::next_vector (16 samples, 0.01%) + + + +doris::RunLengthIntegerReader::_read_delta_values (35 samples, 0.02%) + + + +doris::ReadOnlyFileStream::seek (38 samples, 0.02%) + + + +doris::memory_copy (32 samples, 0.02%) + + + +doris::LargeIntColumnReader::next_vector (41 samples, 0.03%) + + + +doris::ColumnData::_seek_to_block (18 samples, 0.01%) + + + +doris::ReadOnlyFileStream::_assure_data (72 samples, 0.05%) + + + +doris::IntegerColumnWriterWrapper<int, true>::write_batch (376 samples, 0.24%) + + + +doris::RunLengthIntegerWriter::_determined_encoding (765 samples, 0.49%) + + + +doris::OLAPEngine::_build_tablet_info (46 samples, 0.03%) + + + +doris::SegmentReader::seek_to_block (397 samples, 0.26%) + + + +doris::RunLengthIntegerReader::_read_direct_values (14 samples, 0.01%) + + + +doris::memory_copy (166 samples, 0.11%) + + + +__GI___libc_write (325 samples, 0.21%) + + + +doris::OutStream::get_total_buffer_size (2,830 samples, 1.83%) +d.. + + +doris::SegmentWriter::estimate_segment_size (2,868 samples, 1.85%) +d.. + + +doris::ser::read_ints (14 samples, 0.01%) + + + +doris::Reader::_acquire_data_sources (149 samples, 0.10%) + + + +doris::OLAPEngine::push (215 samples, 0.14%) + + + +doris::FieldTypeTraits< (338 samples, 0.22%) + + + +doris::StorageByteBuffer::put (14 samples, 0.01%) + + + +doris::LargeIntColumnReader::next_vector (3,979 samples, 2.57%) +do.. + + +doris::StorageByteBuffer::put (18 samples, 0.01%) + + + +doris::VarStringColumnReader<doris::StringColumnDirectReader>::next_vector (368 samples, 0.24%) + + + +doris::IntegerColumnReaderWrapper<short, true>::seek (17 samples, 0.01%) + + + +memset (23 samples, 0.01%) + + + +doris::ser::read_ints (335 samples, 0.22%) + + + +doris::VectorizedRowBatch::dump_to_row_block (19 samples, 0.01%) + + + +__GI___strncmp_ssse3 (895 samples, 0.58%) + + + +doris::ColumnDataWriter::mem_pool (26 samples, 0.02%) + + + diff --git a/docs/resources/images/cpu-pprof-demo.png b/docs/resources/images/cpu-pprof-demo.png new file mode 100644 index 0000000000..2eb659b1d0 Binary files /dev/null and b/docs/resources/images/cpu-pprof-demo.png differ diff --git a/docs/resources/images/cpu-pprof-demo.svg b/docs/resources/images/cpu-pprof-demo.svg new file mode 100644 index 0000000000..8f00bbb81c --- /dev/null +++ b/docs/resources/images/cpu-pprof-demo.svg @@ -0,0 +1,1879 @@ + + + + + + + + +/home/work/palo2/be/lib/palo_be; 7907 samples + + + +Legend +/home/work/palo2/be/lib/palo_be +Total samples: 7907 +Focusing on: 7907 +Dropped nodes with <= 39 abs(samples) +Dropped edges with <= 7 samples + + + +N1 + +execute_native_thread_routine +0 (0.0%) +of 6794 (85.9%) + + + +N2 + +operator +0 (0.0%) +of 6788 (85.8%) + + + +N1->N2 + + +6788 + + + +N3 + +doris +OLAPEngine +_cumulative_compaction_thread_callback +0 (0.0%) +of 4619 (58.4%) + + + +N2->N3 + + +4619 + + + +N11 + +doris +OLAPEngine +_base_compaction_thread_callback +0 (0.0%) +of 2166 (27.4%) + + + +N2->N11 + + +2166 + + + +N5 + +doris +OLAPEngine +perform_cumulative_compaction +0 (0.0%) +of 4614 (58.4%) + + + +N3->N5 + + +4614 + + + +N4 + +doris +CumulativeCompaction +run +0 (0.0%) +of 4614 (58.4%) + + + +N6 + +doris +CumulativeCompaction +_do_cumulative_compaction +0 (0.0%) +of 4611 (58.3%) + + + +N4->N6 + + +4611 + + + +N5->N4 + + +4614 + + + +N7 + +doris +Merger +merge +17 (0.2%) +of 3969 (50.2%) + + + +N6->N7 + + +2635 + + + +N8 + +doris +Reader +next_row_with_aggregation +5 (0.1%) +of 2812 (35.6%) + + + +N6->N8 + + +1969 + + + +N9 + +doris +ColumnDataWriter +_flush_row_block +0 (0.0%) +of 2751 (34.8%) + + + +N7->N9 + + +2751 + + + +N36 + +doris +Field +cmp +451 (5.7%) + + + +N7->N36 + + +181 + + + +N37 + +generic_compare +422 (5.3%) + + + +N7->N37 + + +89 + + + +N40 + +doris +ColumnDataWriter +_flush_segment_with_verfication +0 (0.0%) +of 329 (4.2%) + + + +N7->N40 + + +329 + + + +N44 + +doris +TypeInfo +cmp +76 (1.0%) +of 316 (4.0%) + + + +N7->N44 + + +62 + + + +N49 + +doris +FieldTypeTraits +cmp +224 (2.8%) + + + +N7->N49 + + +58 + + + +N57 + +doris +ColumnDataWriter +_finalize_segment +0 (0.0%) +of 174 (2.2%) + + + +N7->N57 + + +57 + + + +N66 + +doris +ColumnDataWriter +finalize +0 (0.0%) +of 117 (1.5%) + + + +N7->N66 + + +116 + + + +N17 + +doris +CollectIterator +_merge_next +1 (0.0%) +of 1156 (14.6%) + + + +N8->N17 + + +1156 + + + +N23 + +std +__pop_heap +6 (0.1%) +of 735 (9.3%) + + + +N8->N23 + + +735 + + + +N27 + +doris +CollectIterator +ChildCtxComparator +operator +73 (0.9%) +of 574 (7.3%) + + + +N8->N27 + + +144 + + + +N8->N36 + + +22 + + + +N46 + +doris +RowCursor +full_key_cmp +264 (3.3%) + + + +N8->N46 + + +54 + + + +N50 + +doris +Field +get_field_ptr +213 (2.7%) + + + +N8->N50 + + +48 + + + +N73 + +doris +FieldTypeTraits +copy_without_pool +79 (1.0%) +of 86 (1.1%) + + + +N8->N73 + + +58 + + + +N79 + +doris +TypeInfo +copy_without_pool +39 (0.5%) +of 78 (1.0%) + + + +N8->N79 + + +52 + + + +N10 + +doris +SegmentWriter +write_batch +3 (0.0%) +of 2700 (34.1%) + + + +N9->N10 + + +2700 + + + +N16 + +doris +LargeIntColumnWriter +write_batch +101 (1.3%) +of 1306 (16.5%) + + + +N10->N16 + + +1306 + + + +N10->N36 + + +79 + + + +N39 + +doris +IntegerColumnWriter +write +6 (0.1%) +of 402 (5.1%) + + + +N10->N39 + + +402 + + + +N10->N44 + + +159 + + + +N10->N49 + + +81 + + + +N59 + +doris +ColumnWriter +write +153 (1.9%) + + + +N10->N59 + + +153 + + + +N67 + +doris +ColumnStatistics +add +107 (1.4%) + + + +N10->N67 + + +107 + + + +N78 + +doris +RowBlock +get_row +78 (1.0%) + + + +N10->N78 + + +64 + + + +N14 + +doris +OLAPEngine +perform_base_compaction +0 (0.0%) +of 2165 (27.4%) + + + +N11->N14 + + +2165 + + + +N12 + +doris +BaseCompaction +_do_base_compaction +0 (0.0%) +of 2165 (27.4%) + + + +N12->N7 + + +1334 + + + +N12->N8 + + +828 + + + +N13 + +doris +BaseCompaction +run +0 (0.0%) +of 2165 (27.4%) + + + +N13->N12 + + +2165 + + + +N14->N13 + + +2165 + + + +N15 + +doris +RunLengthIntegerWriter +write +455 (5.8%) +of 1644 (20.8%) + + + +N38 + +doris +RunLengthIntegerWriter +_write_values +9 (0.1%) +of 415 (5.2%) + + + +N15->N38 + + +414 + + + +N42 + +doris +RunLengthIntegerWriter +_determined_encoding +318 (4.0%) +of 319 (4.0%) + + + +N15->N42 + + +316 + + + +N61 + +doris +ser +compute_hists +141 (1.8%) + + + +N15->N61 + + +141 + + + +N68 + +doris +ser +percentile_bits_with_hist +100 (1.3%) + + + +N15->N68 + + +100 + + + +N72 + +std +abs +88 (1.1%) + + + +N15->N72 + + +88 + + + +N16->N15 + + +1201 + + + +N18 + +doris +ColumnData +get_next_block +0 (0.0%) +of 1153 (14.6%) + + + +N17->N18 + + +1151 + + + +N19 + +doris +ColumnData +_get_block_from_reader +2 (0.0%) +of 1149 (14.5%) + + + +N18->N19 + + +653 + + + +N33 + +doris +ColumnData +_get_block +1 (0.0%) +of 551 (7.0%) + + + +N18->N33 + + +498 + + + +N20 + +doris +SegmentReader +get_block +0 (0.0%) +of 1147 (14.5%) + + + +N19->N20 + + +1147 + + + +N21 + +doris +SegmentReader +_load_to_vectorized_row_batch +7 (0.1%) +of 1138 (14.4%) + + + +N20->N21 + + +1138 + + + +N22 + +doris +RunLengthIntegerReader +next +141 (1.8%) +of 737 (9.3%) + + + +N21->N22 + + +721 + + + +N45 + +doris +VarStringColumnReader +next_vector +2 (0.0%) +of 282 (3.6%) + + + +N21->N45 + + +281 + + + +N47 + +doris +ReadOnlyFileStream +read +56 (0.7%) +of 239 (3.0%) + + + +N21->N47 + + +14 + + + +N75 + +doris +StorageByteBuffer +get +50 (0.6%) +of 86 (1.1%) + + + +N21->N75 + + +11 + + + +N77 + +doris +IntegerColumnReaderWrapper +next_vector +81 (1.0%) + + + +N21->N77 + + +80 + + + +N34 + +doris +RunLengthIntegerReader +_read_values +19 (0.2%) +of 527 (6.7%) + + + +N22->N34 + + +525 + + + +N22->N47 + + +26 + + + +N23->N27 + + +430 + + + +N23->N36 + + +59 + + + +N23->N46 + + +89 + + + +N23->N50 + + +74 + + + +N24 + +thread_proxy +0 (0.0%) +of 628 (7.9%) + + + +N25 + +boost +function0 +operator +0 (0.0%) +of 621 (7.9%) + + + +N24->N25 + + +621 + + + +N26 + +doris +OlapScanNode +scanner_thread +0 (0.0%) +of 591 (7.5%) + + + +N25->N26 + + +591 + + + +N26->N8 + + +15 + + + +N29 + +doris +OlapScanner +open +0 (0.0%) +of 566 (7.2%) + + + +N26->N29 + + +566 + + + +N27->N36 + + +104 + + + +N27->N37 + + +74 + + + +N27->N44 + + +95 + + + +N27->N46 + + +121 + + + +N27->N49 + + +51 + + + +N27->N50 + + +56 + + + +N28 + +doris +Reader +init +0 (0.0%) +of 571 (7.2%) + + + +N31 + +doris +Reader +_attach_data_to_merge_set +0 (0.0%) +of 563 (7.1%) + + + +N28->N31 + + +563 + + + +N29->N28 + + +566 + + + +N30 + +doris +ColumnData +prepare_block_read +0 (0.0%) +of 563 (7.1%) + + + +N32 + +doris +ColumnData +_seek_to_row +0 (0.0%) +of 558 (7.1%) + + + +N30->N32 + + +558 + + + +N31->N30 + + +563 + + + +N32->N19 + + +490 + + + +N32->N33 + + +53 + + + +N35 + +doris +VectorizedRowBatch +dump_to_row_block +352 (4.5%) +of 463 (5.9%) + + + +N33->N35 + + +463 + + + +N63 + +doris +memory_copy +131 (1.7%) + + + +N33->N63 + + +36 + + + +N34->N47 + + +11 + + + +N53 + +doris +RunLengthIntegerReader +_read_direct_values +29 (0.4%) +of 205 (2.6%) + + + +N34->N53 + + +201 + + + +N56 + +doris +RunLengthIntegerReader +_read_patched_base_values +34 (0.4%) +of 175 (2.2%) + + + +N34->N56 + + +173 + + + +N80 + +doris +RunLengthIntegerReader +_read_delta_values +59 (0.7%) +of 77 (1.0%) + + + +N34->N80 + + +75 + + + +N35->N63 + + +69 + + + +N70 + +doris +fixed_size_memory_copy +97 (1.2%) + + + +N35->N70 + + +41 + + + +N51 + +doris +OutStream +write +99 (1.3%) +of 210 (2.7%) + + + +N38->N51 + + +10 + + + +N52 + +doris +RunLengthIntegerWriter +_write_direct_values +2 (0.0%) +of 208 (2.6%) + + + +N38->N52 + + +208 + + + +N69 + +doris +RunLengthIntegerWriter +_write_patched_base_values +0 (0.0%) +of 98 (1.2%) + + + +N38->N69 + + +98 + + + +N39->N15 + + +395 + + + +N43 + +doris +SegmentWriter +estimate_segment_size +0 (0.0%) +of 318 (4.0%) + + + +N40->N43 + + +318 + + + +N41 + +doris +OutStream +get_total_buffer_size +320 (4.0%) + + + +N43->N41 + + +316 + + + +N44->N37 + + +198 + + + +N44->N49 + + +34 + + + +N45->N22 + + +16 + + + +N45->N47 + + +93 + + + +N71 + +doris +StringColumnDirectReader +next_vector +88 (1.1%) +of 90 (1.1%) + + + +N45->N71 + + +90 + + + +N45->N75 + + +51 + + + +N54 + +doris +ReadOnlyFileStream +_assure_data +112 (1.4%) +of 184 (2.3%) + + + +N47->N54 + + +161 + + + +N74 + +doris +ReadOnlyFileStream +FileCursor +read +1 (0.0%) +of 86 (1.1%) + + + +N47->N74 + + +20 + + + +N48 + +doris +ser +read_ints +234 (3.0%) + + + +N60 + +doris +StorageByteBuffer +put +114 (1.4%) +of 145 (1.8%) + + + +N51->N60 + + +73 + + + +N52->N51 + + +85 + + + +N55 + +doris +ser +write_ints +176 (2.2%) + + + +N52->N55 + + +102 + + + +N52->N60 + + +16 + + + +N53->N47 + + +47 + + + +N53->N48 + + +117 + + + +N54->N74 + + +66 + + + +N56->N47 + + +31 + + + +N56->N48 + + +105 + + + +N58 + +doris +SegmentWriter +finalize +0 (0.0%) +of 157 (2.0%) + + + +N57->N58 + + +157 + + + +N64 + +doris +OutStream +write_to_file +2 (0.0%) +of 129 (1.6%) + + + +N58->N64 + + +129 + + + +N60->N60 + + +33 + + + +N60->N63 + + +10 + + + +N60->N70 + + +20 + + + +N62 + +0x00007f878622feed +139 (1.8%) + + + +N64->N62 + + +96 + + + +N65 + +LZ4_compress_generic +124 (1.6%) + + + +N66->N57 + + +117 + + + +N69->N51 + + +25 + + + +N69->N55 + + +63 + + + +N69->N60 + + +10 + + + +N76 + +0x00007f878622e1c3 +83 (1.0%) + + + +N74->N76 + + +83 + + + +N75->N63 + + +8 + + + +N75->N70 + + +28 + + + +N79->N73 + + +27 + + + +N80->N48 + + +11 + + + diff --git a/docs/resources/images/perf-report-demo.png b/docs/resources/images/perf-report-demo.png new file mode 100644 index 0000000000..e702f08c7d Binary files /dev/null and b/docs/resources/images/perf-report-demo.png differ