database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,239 Commits 1 Branch 0 Tags
4e4f3d204ecf30166e177a02ef38ec3e007e7dfd
Commit Graph

466 Commits

Author SHA1 Message Date
Mingyu Chen
ceef9ee123 [feature](serde) support presto compatible output format (#37039) (#37253) 
bp #37039
 2024-07-04 13:56:05 +08:00
walter
b3dcfae864 [chore](be) Improve ingesting binlog error checking (#36596) 
Cherry-pick #36487
 2024-06-20 20:15:26 +08:00
Kaijie Chen
612f2ae961 [feature](api) add BE HTTP /api/load_streams (#36312) (#36338) 
cherry-pick #36312
 2024-06-16 22:09:04 +08:00
lihangyu
0b28420e1c [pick](Variant) make remote schema fetch rpc timeout configurable (#35296) (#36174) 2024-06-12 19:51:53 +08:00
yiguolei
c033c71aed [enhancement](cancel) add cancel query ability on be 2024-06-03 11:43:57 +08:00
Pxl
0f7cc5fddc [Bug](backend-options) fix use after free on BackendOptions::get_local_backend() (#35634) 
## Proposed changes

```cpp
==1825462==ERROR: AddressSanitizer: heap-use-after-free on address 0x5030012ea880 at pc 0x555cc645e1aa bp 0x7fe4a9407e20 sp 0x7fe4a94075e0
READ of size 22 at 0x5030012ea880 thread T2250 (REPORT_DISK_STA)
    #0 0x555cc645e1a9 in __asan_memcpy (/mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be+0x22cb01a9) (BuildId: 76fba423c52f4da5)
    #1 0x555cc64b6990 in std::char_traits<char>::copy(char*, char const*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/char_traits.h:445:33
    #2 0x555cc64b68a1 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_S_copy(char*, char const*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:420:4
    #3 0x555cc64c526e in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.tcc:291:6
    #4 0x555cc64c5070 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:1596:8
    #5 0x555cc64b1d9c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:802:15
    #6 0x555ccb05a722 in doris::TBackend::TBackend(doris::TBackend const&) /mnt/disk1/xiaolei/incubator-doris/gensrc/build/gen_cpp/Types_types.cpp:5755:8
    #7 0x555cc9c9234e in doris::BackendOptions::get_local_backend() /mnt/disk1/xiaolei/incubator-doris/be/src/service/backend_options.cpp:79:12
    #8 0x555cc6586e5a in doris::report_disk_callback(doris::StorageEngine&, doris::TMasterInfo const&) /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:1021:27
    #9 0x555cc64f2a0a in doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/agent_server.cpp:190:133
    #10 0x555cc64f2994 in void std::__invoke_impl<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>(std::__invoke_other, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    #11 0x555cc64f2944 in std::enable_if<is_invocable_r_v<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>, void>::type std::__invoke_r<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>(doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2
    #12 0x555cc64f280c in std::_Function_handler<void (), doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
    #13 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
    #14 0x555cc65b01f9 in doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:689:13
    #15 0x555cc65afcd4 in void std::__invoke_impl<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(std::__invoke_other, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    #16 0x555cc65afc74 in std::enable_if<is_invocable_r_v<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>, void>::type std::__invoke_r<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2
    #17 0x555cc65afa6c in std::_Function_handler<void (), doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
    #18 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
    #19 0x555cca138dab in doris::Thread::supervise_thread(void*) /mnt/disk1/xiaolei/incubator-doris/be/src/util/thread.cpp:498:5
    #20 0x555cc645de0a in asan_thread_start(void*) crtstuff.c
    #21 0x7fef061601c9 in start_thread (/lib64/libpthread.so.0+0x81c9) (BuildId: 823fccea3475e5870a4167dfe47df20e53222db0)
    #22 0x7fef06b4fe72 in clone (/lib64/libc.so.6+0x39e72) (BuildId: ec3d7025354f1f1985831ff08ef0eb3b50aefbce)

0x5030012ea880 is located 0 bytes inside of 31-byte region [0x5030012ea880,0x5030012ea89f)
freed by thread T2249 (REPORT_TASK-182) here:
    #0 0x555cc649d43d in operator delete(void*) (/mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be+0x22cef43d) (BuildId: 76fba423c52f4da5)
    #1 0x555cc64b6bbc in std::__new_allocator<char>::deallocate(char*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/new_allocator.h:168:2
    #2 0x555cc64b6b5d in std::allocator<char>::deallocate(char*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/allocator.h:210:25
    #3 0x555cc64b6b5d in std::allocator_traits<std::allocator<char>>::deallocate(std::allocator<char>&, char*, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/alloc_traits.h:516:13
    #4 0x555cc64b6b5d in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_destroy(unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:289:9
    #5 0x555cc64b6a56 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_dispose() /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:283:4
    #6 0x555cc64c51ee in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.tcc:285:8
    #7 0x555cc64c5070 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:1596:8
    #8 0x555cc64b1d9c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:802:15
    #9 0x555ccb058510 in doris::TBackend::__set_host(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/incubator-doris/gensrc/build/gen_cpp/Types_types.cpp:5584:14
    #10 0x555cc9c92239 in doris::BackendOptions::get_local_backend() /mnt/disk1/xiaolei/incubator-doris/be/src/service/backend_options.cpp:75:14
    #11 0x555cc6585b9d in doris::report_task_callback(doris::TMasterInfo const&) /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:1004:27
    #12 0x555cc64f24b9 in doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/agent_server.cpp:187:112
    #13 0x555cc64f2474 in void std::__invoke_impl<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&>(std::__invoke_other, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    #14 0x555cc64f2424 in std::enable_if<is_invocable_r_v<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&>, void>::type std::__invoke_r<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&>(doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2
    #15 0x555cc64f22ec in std::_Function_handler<void (), doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_20>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
    #16 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
    #17 0x555cc65b01f9 in doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:689:13
    #18 0x555cc65afcd4 in void std::__invoke_impl<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(std::__invoke_other, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    #19 0x555cc65afc74 in std::enable_if<is_invocable_r_v<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>, void>::type std::__invoke_r<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2
    #20 0x555cc65afa6c in std::_Function_handler<void (), doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
    #21 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
    #22 0x555cca138dab in doris::Thread::supervise_thread(void*) /mnt/disk1/xiaolei/incubator-doris/be/src/util/thread.cpp:498:5
    #23 0x555cc645de0a in asan_thread_start(void*) crtstuff.c

previously allocated by thread T2250 (REPORT_DISK_STA) here:
    #0 0x555cc649cbdd in operator new(unsigned long) (/mnt/disk1/xiaolei/incubator-doris/output/be/lib/doris_be+0x22ceebdd) (BuildId: 76fba423c52f4da5)
    #1 0x555cc64b684e in std::__new_allocator<char>::allocate(unsigned long, void const*) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/new_allocator.h:147:27
    #2 0x555cc64b67b0 in std::allocator<char>::allocate(unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/allocator.h:198:32
    #3 0x555cc64b67b0 in std::allocator_traits<std::allocator<char>>::allocate(std::allocator<char>&, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/alloc_traits.h:482:20
    #4 0x555cc64b67b0 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_S_allocate(std::allocator<char>&, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:126:16
    #5 0x555cc64b63b1 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_create(unsigned long&, unsigned long) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.tcc:155:14
    #6 0x555cc64c51de in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.tcc:284:24
    #7 0x555cc64c5070 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:1596:8
    #8 0x555cc64b1d9c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/basic_string.h:802:15
    #9 0x555ccb058510 in doris::TBackend::__set_host(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /mnt/disk1/xiaolei/incubator-doris/gensrc/build/gen_cpp/Types_types.cpp:5584:14
    #10 0x555cc9c92239 in doris::BackendOptions::get_local_backend() /mnt/disk1/xiaolei/incubator-doris/be/src/service/backend_options.cpp:75:14
    #11 0x555cc6586e5a in doris::report_disk_callback(doris::StorageEngine&, doris::TMasterInfo const&) /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:1021:27
    #12 0x555cc64f2a0a in doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/agent_server.cpp:190:133
    #13 0x555cc64f2994 in void std::__invoke_impl<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>(std::__invoke_other, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    #14 0x555cc64f2944 in std::enable_if<is_invocable_r_v<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>, void>::type std::__invoke_r<void, doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&>(doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2
    #15 0x555cc64f280c in std::_Function_handler<void (), doris::AgentServer::start_workers(doris::StorageEngine&, doris::ExecEnv*)::$_21>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
    #16 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
    #17 0x555cc65b01f9 in doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0::operator()() const /mnt/disk1/xiaolei/incubator-doris/be/src/agent/task_worker_pool.cpp:689:13
    #18 0x555cc65afcd4 in void std::__invoke_impl<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(std::__invoke_other, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:61:14
    #19 0x555cc65afc74 in std::enable_if<is_invocable_r_v<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>, void>::type std::__invoke_r<void, doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&>(doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/invoke.h:111:2
    #20 0x555cc65afa6c in std::_Function_handler<void (), doris::ReportWorker::ReportWorker(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, doris::TMasterInfo const&, int, std::function<void ()>)::$_0>::_M_invoke(std::_Any_data const&) /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:290:9
    #21 0x555cc6612ef2 in std::function<void ()>::operator()() const /mnt/disk1/xiaolei/ldb_17/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/bits/std_function.h:591:9
    #22 0x555cca138dab in doris::Thread::supervise_thread(void*) /mnt/disk1/xiaolei/incubator-doris/be/src/util/thread.cpp:498:5
    #23 0x555cc645de0a in asan_thread_start(void*) crtstuff.c

```
 2024-05-30 19:59:37 +08:00
Kaijie Chen
7058b31edd [fix](move-memtable) clear load streams before shutdown SegmentFileWriterThreadPool (#35217) 2024-05-28 13:12:03 +08:00
Sun Chenyang
6e17dc1e87 (cherry-pick)[branch-2.1] add calc tablet file crc and fix single compaction test #33076 #34915 (#35215) 
* [fix](compaction test) show single replica compaction status and fix test (#33076)
* [improve](http action) add http interface to calculate the crc of all files in tablet (#34915)
 2024-05-26 17:15:09 +08:00
HHoflittlefish777
7ba66c5890 [branch-2.1](routine-load) do not schedule task when there is no data (#34654) 2024-05-11 11:01:18 +08:00
lihangyu
853dbdcb00 [Feature](PreparedStatement) implement general server side prepared (#33807) 2024-05-10 22:10:11 +08:00
yiguolei
5271042a7d [bugfix](gccompile) fix gcc compile error (#34546) 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-05-10 22:06:00 +08:00
yiguolei
32cbd4a583 [chore](status) unify error code between thrift,pb, status.h (#34397) 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-05-10 14:41:01 +08:00
zxealous
9aa08d8deb [improve](disk) Not add disk path to broken list if check status is not IO_ERROR (#34111) 2024-04-26 07:44:12 +08:00
StarryVerse
4f6b9db7a7 Update doris_main.cpp (#34128) 
* Update doris_main.cpp

Log(FATAL) introduces a core dump, which is confusing for users. We should print error msg and exit without a core dump.

* Update doris_main.cpp
 2024-04-26 07:43:40 +08:00
deardeng
a050513c91 [Fix](clean trash) Fix clean trash use agent task (#33912) (#33972) 
* [Fix](clean trash) Fix clean trash use agent task (#33912)

* add .h
 2024-04-22 17:14:21 +08:00
TengJianPing
615765c1c0 [improvement](spill) improve spill directory and fix bugs (#33900) 
* [improvement](spill) improve spill directory and fix bugs

* fix
 2024-04-22 11:28:22 +08:00
Tiewei Fang
315f6e44c2 [Branch-2.1](Outfile) Fixed the problem that the concurrent Outfile wrote multiple Success files (#33870) 
backport: #33016
 2024-04-19 12:09:53 +08:00
HHoflittlefish777
e0ec2da29b [fix](routine-load) fix get kafka offset timeout may too long (#33502) 2024-04-17 23:42:12 +08:00
qiye
46a258dc85 [improvement](binlog)Support inverted index format v2 in CCR (#33415) 2024-04-17 23:42:12 +08:00
yiguolei
af95302088 fix compile 2024-04-11 13:10:24 +08:00
Xinyi Zou
69fc8cf06d [branch-2.1](memory) Fix rowid storage reader memory tracker (#33521) 
fix:
F20240411 10:26:06.693233 1368925 thread_context.h:293] __builtin_unreachable, If you crash here, it means that SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER are not used correctly. starting position of each thread is expected to use SCOPED_ATTACH_TASK to bind a MemTrackerLimiter belonging to Query/Load/Compaction/Other Tasks, otherwise memory alloc using Doris Allocator in the thread will crash. If you want to switch MemTrackerLimiter during thread execution, please use SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER, do not repeat Attach. Of course, you can modify enable_memory_orphan_check=false in be.conf to avoid this crash.
*** Check failure stack trace: ***
    @     0x562d9b5aa6a6  google::LogMessage::SendToLog()
    @     0x562d9b5a70f0  google::LogMessage::Flush()
    @     0x562d9b5aaee9  google::LogMessageFatal::~LogMessageFatal()
    @     0x562d7ebd1b7e  doris::thread_context()
    @     0x562d7ec203b8  Allocator<>::sys_memory_check()
    @     0x562d7ec255a3  Allocator<>::memory_check()
    @     0x562d7ec274a1  Allocator<>::alloc_impl()
    @     0x562d7ec27227  Allocator<>::alloc()
    @     0x562d67a12207  doris::vectorized::PODArrayBase<>::alloc<>()
    @     0x562d67a11fde  doris::vectorized::PODArrayBase<>::realloc<>()
    @     0x562d67a11e26  doris::vectorized::PODArrayBase<>::reserve<>()
    @     0x562d77331ee3  doris::vectorized::ColumnVector<>::reserve()
    @     0x562d7e64328e  doris::vectorized::ColumnNullable::reserve()
    @     0x562d7ec79a84  doris::vectorized::Block::Block()
    @     0x562d6b86b81b  doris::PInternalServiceImpl::_multi_get()
    @     0x562d6b8a4a07  doris::PInternalServiceImpl::multiget_data()::$_0::operator()()
 2024-04-11 13:10:24 +08:00
Pxl
3081fc584d [Improvement](runtime-filter) support sync join node build side's size to init bloom runtime filter (#32180) 
support sync join node build side's size to init bloom runtime filter
 2024-04-11 09:31:50 +08:00
TengJianPing
0e262ba0e4 [improvement](spill) improve cancel of spill and improve log printing (#33229) 
* [improvement](spill) improve cancel of spill and improve log printing

* fix
 2024-04-10 16:23:20 +08:00
Xinyi Zou
cf7595d423 [opt](memory) Optimize mem tracker accuracy (#32039) (#33140) 2024-04-10 11:42:19 +08:00
Xinyi Zou
06e5c6c966 [fix](grace-exit) Stop incorrectly of reportwork cause heap use after free #32929 2024-04-10 11:34:28 +08:00
Mingyu Chen
95da52b9d8 [fix](avro) avoid BE crash if avro scanner's dependency jars is mssing (#33031) 
1. Check the return value of avro reader's init_fetch_table_schema_reader()
2. Also fix a bug but the parse exception of Nereids may suppress the real exception from old planner
    It will result unable to see the real error msg.
 2024-04-07 22:17:16 +08:00
lihangyu
92d7333810 [Fix](point query) avoid nullptr in _block_pool (#33120) 
`resize` will make nullptrs in _block_pool if _block_pool.size() < s_preallocted_blocks_num
 2024-04-07 13:02:37 +08:00
lihangyu
62c7d0a421 [Fix](point query) add query options for short circuit queries (#32530) (#32684) 
Some options like `be_exec_version` needed for functions
 2024-03-22 18:03:18 +08:00
Gabriel
ab512f935c [pipelineX](api) Add api for long-running tasks (#32459) 2024-03-21 14:07:24 +08:00
Gabriel
4bf5a21ba3 [pipelineX](cancel) Remove lock for mapping query ctx to fragment (#32346) 2024-03-21 14:07:23 +08:00
yiguolei
85b2c42f76 [Enhancement](jdbc catalog) Add a property to test the connection when creating a Jdbc catalog (#32125) (#32531) 2024-03-21 14:05:59 +08:00
airborne12
ecadb60bcd [Pick 2.1](inverted index) support inverted index format v2 (#30145) (#32418) 2024-03-19 08:11:33 +08:00
Xinyi Zou
7b74b199a5 [fix](memory) Fix LRU cache deleter and memory tracking (#32080) 
In order to add common code to the value deleter of LRU cache, let all lru cache values inherit from LRUCacheValueBase class and tracking memory in destructor.
 2024-03-15 17:57:58 +08:00
qiye
5f125bbaaa [improvement](binlog)Support inverted index in CCR (#31743) (#32101) 2024-03-12 15:34:08 +08:00
TengJianPing
3358f76a7f [feature](spill) Implement spill to disk for hash join, aggregation and sort for pipelineX (#31910) 
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
 2024-03-12 14:12:09 +08:00
lihangyu
0da010603e [Improve](TabletSchemaCache) reduce duplicated memory consumption for column name and column path (#31141) 
Both could be reference to related field in TabletColumn.And use shared_ptr for TabletColumn in TabletSchema for later memory reuse
 2024-03-09 19:44:42 +08:00
zhangstar333
c40c16b8b3 [improve](conf)refactor fuzzy mode in BE (#31412) 
refactor the code of fuzzy in BE, and will be add more variables in it, then could test case at different mode.
 2024-02-29 19:51:07 +08:00
zclllyybb
b177b26d39 [branch-2.1](tracing) Pick pipeline tracing and relative bugfix (#31367) 
* [Feature](pipeline) Trace pipeline scheduling (part I) (#31027)

* [fix](compile) Fix performance compile fail #31305

* [fix](compile) Fix macOS compilation issues for PURE macro and CPU core identification (#31357)

* [fix](compile) Correct PURE macro definition to fix compilation on macOS

* 2

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
 2024-02-29 08:42:35 +08:00
HappenLee
52c45e38af [Refactor](RF) refactor the profile of rf and pipeline-x support local ignore (#31287) 
* [Refactor](RF) refactor the profile of rf and pipeline-x support local ignore

* fix local merge filter
 2024-02-23 19:05:06 +08:00
Xinyi Zou
1abe9d4384 [fix](memory) Fix LRU cache stale sweep (#31122) 
Remove LRUCacheValueBase, put last_visit_time into LRUHandle, and automatically update timestamp to last_visit_time during cache insert and lookup.

Do not rely on external modification of last_visit_time, which is often forgotten.
 2024-02-21 17:01:29 +08:00
Mingyu Chen
fc53c7210b [fix](chmod) change chmod to filesystem::permission to avoid race condition (#31032) 2024-02-18 11:50:16 +08:00
yiguolei
09ef78402e fix build error 2024-02-05 22:15:17 +08:00
HappenLee
be31b8dc61 [Refactor](exchange) remove unless code in exchange and opt some code (#30813) 2024-02-05 21:59:52 +08:00
py023
4b42156fc0 [chore](clang-tidy): add bugprone linters (#29521) 
This PR introduces 4 bugprone linter rules to .clang-tidy, these linters found some bugs in #28965. This PR also add some comments to mute false positive reports.
 2024-02-05 21:58:08 +08:00
Kaijie Chen
5a0764b288 [log](rpc) print log when offer_failed in internal service (#30454) 2024-02-01 19:01:08 +08:00
zclllyybb
24ed3e4103 [Fix](Expr&code-style) check prepare&open before every VExpr execute (#26673) 2024-01-23 10:09:54 +08:00
Kaijie Chen
620cfc3cd7 [fix](move-memtable) set idle timeout equal to load timeout (#29839) 2024-01-16 18:33:51 +08:00
yiguolei
c58d18147b [refactor](resultwriter) rename append_block to write method is more meaningful (#29635) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-08 00:00:33 +08:00
huanghaibin
0b731800a0 [enhancement](group_commit) refector wal manager code (#29560) 2024-01-07 18:54:41 +08:00
AlexYue
d61fec3ff8 (feature)(LOG) Support to change be vlog level dynamically using http (#28785) 2024-01-06 20:12:35 +08:00