database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
11,088 Commits 1 Branch 0 Tags
4faee4d8fd72d1edf4bc681abe5c22fb572cfbcb
Commit Graph

11088 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Qi Chen
4faee4d8fd [Fix](multi-catalog) Fix be crashed when query hive table after schema changed(new column added). (#20537) 
Fix be crashed when query hive table after schema changed(new column added).

Regression Test: test_hive_schema_evolution.groovy
 2023-06-08 18:10:36 +08:00
mch_ucchi
41d7c535f2 [fix](regression-test) add sync after insert into table for nereids case (#20516) 2023-06-08 17:52:36 +08:00
Kaijie Chen
3054574bc1 [fix](load) fix ctx nullptr core in flush_single_memtable (#20573) 2023-06-08 17:40:02 +08:00
Pxl
a56449f86e [Bug](Agg-state) try to make test_agg_state stable (#20574) 
try to make test_agg_state stable
 2023-06-08 17:17:51 +08:00
plat1ko
a68fc551f0 [bug](cooldown) Fix async_write_cooldown_meta and snapshot cooldowned version not continuous bug (#20437) 2023-06-08 15:35:35 +08:00
Pxl
5fe7106b83 [Bug](planner) fix pre condition check fail on max(null) (#20509) 
fix pre condition check fail on max(null)
 2023-06-08 14:49:52 +08:00
Pxl
22985af4d7 [Bug](pipeline) set SourceState to MORE_DATA when UnionSourceOperator have const_expr/data_queue->remaining_has_data (#20557) 
set SourceState to MORE_DATA when UnionSourceOperator have const_expr/data_queue->remaining_has_data
 2023-06-08 14:47:35 +08:00
yagagagaga
7f39d58e92 [typo](docs) fix some description error about segcompaction_small_threshold (#20421) 2023-06-08 14:31:04 +08:00
Xiangyu Wang
6702b6ca57 [Fix](hive-catalog) Fallback to refresh catalog when hms events are missing (#20227) 
This error can not be recovered (the relevant events in hms may have been deleted and can not recovered), so we need a fallback.
 2023-06-08 13:43:10 +08:00
wudi
ee4c041444 [doc](flink) add flink delete column from kafka specified columns (#20545) 2023-06-08 13:42:27 +08:00
FreeOnePlus
f429276863 [typo(docs)Change docker docs DorisImage version (#20580) 2023-06-08 13:41:48 +08:00
FreeOnePlus
43811ea989 [improvement](docker)Change docker shell DorisImage version (#20581) 2023-06-08 13:41:17 +08:00
lihangyu
24fb05ec83 [Bug](row-store) Fix row store with materialize index (#20356) 
If a query hits a materialized view that has row storage enabled, but the row storage column is not present in the materialized view, it will result in a query crash. Therefore, it is necessary to include the row storage column when creating the materialized view, and serialize the row storage column during the execution of SchemaChange.
 2023-06-08 10:55:22 +08:00
zhangdong
46c68d11aa [feature-wip](MTMV) Sync finish status only for tasks (#20441) 
MTMV tasks keep finish status only to reduce the loss caused by logging.
After changes, unfinished tasks will be lost directly when FE master restarts.
 2023-06-08 10:46:25 +08:00
starocean999
d2d6ce5d0b [fix](nereids) add push down filter and project through cte anchor rules (#20547) 
we should not plan any Filter or Project above CteAnchor, because there are project or filter under anchor sometimes.
and the whole plan can not translate to a valid plan for BE.
 2023-06-08 10:34:42 +08:00
Gabriel
325ddab34e [conf](pipeline) turn pipeline on by default (#20458) 2023-06-08 09:20:51 +08:00
yujun
f154319a49 [fix](load tablet) Fix restart slowly due to load deleted tablet error (#20552) 2023-06-08 09:16:33 +08:00
davidshtian
d4b568d5ab [typo](doc)Add Broker installation and upgrade to 1.2.2 (#17878) 
Add Broker installation and upgrade 1.2.2 docker image version
 2023-06-07 23:32:17 +08:00
yagagagaga
37d636cd11 [typo](docs) Update profile screenshot. (#20255) 2023-06-07 23:23:10 +08:00
yujun
92577f45d3 [fix] (recover) fix can not recover a BE's tablet after deleting its data directory manual (#20273) (#20274) 2023-06-07 22:27:50 +08:00
bobhan1
187bf14d81 [feature-wip](auto-inc)(step-1) add syntax support for duplicate table (#20284) 
Co-authored-by: yifeng <cnissnzg@126.com>
 2023-06-07 22:01:28 +08:00
Jack Drogon
03cb69c0ee [feature](backup-restore) Add local backup/restore not upload/download by broker (#20492) 2023-06-07 21:35:15 +08:00
zhannngchen
53970192aa [fix](regression) unique_with_mow_p2/test_pk_uk_case (#20497) 2023-06-07 21:34:34 +08:00
airborne12
296ca1d9dd [Fix](inverted index) if range query exceeds CLucene limits, downgrade it from inverted index (#20528) 
CLucene may throw CL_ERR_TooManyClauses when a range query hits too many terms.
In this situation, we have to downgrade from inverted index.
 2023-06-07 20:07:48 +08:00
zhangdong
809bb46518 [doc](multi-catalog)huawei mrs event sync doc (#20466) 2023-06-07 20:05:12 +08:00
HHoflittlefish777
2db900b775 [fix](lazy_open) fix lazy open null point (#20540) 2023-06-07 17:56:31 +08:00
zhengyu
09344eaab5 [feature](load) introduce single-stream-multi-table load (#20006) 
For routine load (kafka load), user can produce all data for different
table into single topic and doris will dispatch them into corresponding
table.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
 2023-06-07 17:55:25 +08:00
Pxl
fbbf4c420e [Bug](Agg-State) fix agg state function get wrong input argument list (#20546) 
fix agg state function get wrong input argument list
 2023-06-07 17:32:48 +08:00
HappenLee
d00b7ad04b [Opt](performance) opt the outer join for nested loop join (#20524) 2023-06-07 17:31:36 +08:00
TengJianPing
841094960f [fix](olapscanner) fix coredump caused by concurrent acccess of olap scan node _conjuncts (#20534) 
=3073084==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60601897db80 at pc 0x55b2c993666e bp 0x7d1fbbfb66b0 sp 0x7d1fbbfb66a8
READ of size 8 at 0x60601897db80 thread T610 (_scanner_scan)
    #0 0x55b2c993666d in std::__shared_ptr<doris::vectorized::VExprContext, (__gnu_cxx::_Lock_policy)2>::get() const /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291:16
    #1 0x55b2dae86ec5 in doris::vectorized::VExprContext::clone(doris::RuntimeState*, std::shared_ptr<doris::vectorized::VExprContext>&) /mnt/disk2/tengjianping/doris-master/be/src/vec/exprs/vexpr_context.cpp:98:5
    #2 0x55b2e757b6d8 in doris::vectorized::VScanner::prepare(doris::RuntimeState*, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext>>> const&) /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/vscanner.cpp:47:13
    #3 0x55b2e78e8155 in doris::vectorized::NewOlapScanner::init() /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/new_olap_scanner.cpp:109:5
    #4 0x55b2e7551c81 in doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, std::shared_ptr<doris::vectorized::VScanner>) /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/scanner_scheduler.cpp:279:27
    #5 0x55b2e7554d5e in doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()::operator()() const /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/scanner_scheduler.cpp:202:31
    #6 0x55b2e7554c14 in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #7 0x55b2e7554bb4 in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&>(doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
    #8 0x55b2e7554a1c in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()>::_M_invoke(std::_Any_data const&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
    #9 0x55b2c80f2cd2 in std::function<void ()>::operator()() const /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
    #10 0x55b2e755f3e4 in doris::PriorityWorkStealingThreadPool::work_thread(int) /mnt/disk2/tengjianping/doris-master/be/src/util/priority_work_stealing_thread_pool.hpp:135:17
    #11 0x55b2e7563c72 in void std::__invoke_impl<void, void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>(std::__invoke_memfun_deref, void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14
    #12 0x55b2e7563b44 in std::__invoke_result<void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>::type std::__invoke<void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>(void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    #13 0x55b2e7563b14 in decltype(std::__invoke((*this)._M_pmf, std::forward<doris::PriorityWorkStealingThreadPool*&>(fp), std::forward<int&>(fp))) std::_Mem_fn_base<void (doris::PriorityWorkStealingThreadPool::*)(int), true>::operator()<doris::PriorityWorkStealingThreadPool*&, int&>(doris::PriorityWorkStealingThreadPool*&, int&) const /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:131:11
    #14 0x55b2e7563ae4 in void std::__invoke_impl<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>(std::__invoke_other, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #15 0x55b2e7563a54 in std::enable_if<is_invocable_r_v<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>, void>::type std::__invoke_r<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>(std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
    #16 0x55b2e75639c3 in void std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>::__call<void, 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:570:11
    #17 0x55b2e756382d in void std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>::operator()<>() /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:629:17
    #18 0x55b2e7563744 in void std::__invoke_impl<void, std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>(std::__invoke_other, std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>&&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #19 0x55b2e7563704 in std::__invoke_result<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>::type std::__invoke<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>(std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>&&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    #20 0x55b2e75636dc in void std:🧵:_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:253:13
    #21 0x55b2e75636b4 in std:🧵:_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>::operator()() /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:260:11
    #22 0x55b2e7563638 in std:🧵:_State_impl<std:🧵:_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>>::_M_run() /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13
    #23 0x55b2eb41d0ef in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18
    #24 0x7f1dfd4e1179 in start_thread pthread_create.c
    #25 0x7f1dfdd7bdf2 in clone (/lib64/libc.so.6+0xfcdf2) (BuildId: 20ee73ce1b6ac38a52440bab82ec7e28f0f5c5b9)
 2023-06-07 17:00:29 +08:00
bingquanzhao
7ead287405 [typo](doc)Fixed some date function documentation descriptions (#20539) 2023-06-07 16:30:19 +08:00
catpineapple
40205503d7 [doc](fix) update cold hot separation chinese doc (#20542) 2023-06-07 16:29:57 +08:00
zhangdong
c910e9b78b [doc](disk)fix disk capacity doc error (#20506) 2023-06-07 15:20:04 +08:00
jakevin
6b325a8458 [fix](Nereids): union output can be Alias<Slot> (#20532) 2023-06-07 15:11:07 +08:00
yuxuan-luo
732526ba58 [doc](multi-catalog) paimon catalog doc (#20503) 2023-06-07 14:21:07 +08:00
minghong
a4acd2c06e [chore](gitignore) Update gitignore to skip: tpcds tools and third party zip (#20529) 
* upate git ignore: tpcds tools, third party zip
 2023-06-07 13:02:10 +08:00
Pxl
36216f0925 [Bug](Agg-State) fix coredump when state combinator input const column (#20510) 
fix coredump when state combinator input const column
 2023-06-07 11:28:55 +08:00
starocean999
cd70c37402 [fix](nereids) filter and project node should be pushed down through cte (#20508) 
1.move PushdownFilterThroughCTEAnchor and PushdownProjectThroughCTEAnchor into PUSH_DOWN_FILTERS rule set
2.move PushdownFilterThroughProject before MergeProjectPostProcessor
 2023-06-07 10:36:32 +08:00
AKIRA
b83039de76 [fix](stats) Make alter column stats no forward (#20501) 
For test convenient, since daily regression tests queries would be sent any FE rather than master only.
 2023-06-07 10:14:44 +08:00
Jerry Hu
03a679e33a [improvement](sink) reuse rows buffer in msyql_result_writer (#20482) 
Creating a rows buffer for each block can impact non-negligible performance.
So it is necessary to reuse the rows buffer.

Test with a total of 1.7M rows, the AppendBatchTime reduced from 500ms to 280ms.
 2023-06-07 10:09:32 +08:00
zhangstar333
35a2be1074 [refactor](agg_state) refactor agg_state type to support fixed length object type (#20370) 
before the agg_state type only support with datatype string,
But with some agg functions, eg: avg,sum,mix...
those functions need serialize type is fixed length object type
 2023-06-07 10:05:00 +08:00
Super .Wein（星痕）
105d9a9513 [typo](doc) Fix a wrongly written in the release note for 1.2.0 version (#20496) 2023-06-07 09:39:42 +08:00
Jerry Hu
49f8f20fb1 [fix](regex) String with Chinese characters matching failed (#20493) 2023-06-07 07:27:47 +08:00
zhangstar333
3691372054 [bug](table_function) fix table function node forget to call open function of expr (#20495) 2023-06-07 07:26:50 +08:00
wangbo
3aa0c2bdbf [Bug](memleak) Fix emptyoperator may cause node not close (#20525) 2023-06-07 01:27:13 +08:00
yuxuan-luo
b65094c8df [Improvement](multi-catalog) paimon supports projection push down (#20522) 
Co-authored-by: hugoluo <hugoluo@tencent.com>
 2023-06-07 00:39:08 +08:00
gnehil
880e2d8373 [typo](doc) update spark connnector version compatibility instructions (#20477) 2023-06-06 23:33:27 +08:00
bingquanzhao
43ae2c59c3 [typo](doc) Fixed some description of date_format function documentation (#20504) 2023-06-06 23:32:34 +08:00
Hong Liu
5a749e6f4d [doc](catalog-hive) Add the property of hive catalog with kerberos. (#20502) 
Co-authored-by: smallhibiscus <844981280>
 2023-06-06 23:30:19 +08:00
AlexYue
c991249360 [enhancement](cooldown) use cooldown replica first when generating scan node (#20384) 2023-06-06 22:15:49 +08:00