database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
11,064 Commits 1 Branch 0 Tags
809bb4651895c3d951f4b98fc7366aa426f169c4
Commit Graph

11064 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
zhangdong
809bb46518 [doc](multi-catalog)huawei mrs event sync doc (#20466) 2023-06-07 20:05:12 +08:00
HHoflittlefish777
2db900b775 [fix](lazy_open) fix lazy open null point (#20540) 2023-06-07 17:56:31 +08:00
zhengyu
09344eaab5 [feature](load) introduce single-stream-multi-table load (#20006) 
For routine load (kafka load), user can produce all data for different
table into single topic and doris will dispatch them into corresponding
table.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
 2023-06-07 17:55:25 +08:00
Pxl
fbbf4c420e [Bug](Agg-State) fix agg state function get wrong input argument list (#20546) 
fix agg state function get wrong input argument list
 2023-06-07 17:32:48 +08:00
HappenLee
d00b7ad04b [Opt](performance) opt the outer join for nested loop join (#20524) 2023-06-07 17:31:36 +08:00
TengJianPing
841094960f [fix](olapscanner) fix coredump caused by concurrent acccess of olap scan node _conjuncts (#20534) 
=3073084==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60601897db80 at pc 0x55b2c993666e bp 0x7d1fbbfb66b0 sp 0x7d1fbbfb66a8
READ of size 8 at 0x60601897db80 thread T610 (_scanner_scan)
    #0 0x55b2c993666d in std::__shared_ptr<doris::vectorized::VExprContext, (__gnu_cxx::_Lock_policy)2>::get() const /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291:16
    #1 0x55b2dae86ec5 in doris::vectorized::VExprContext::clone(doris::RuntimeState*, std::shared_ptr<doris::vectorized::VExprContext>&) /mnt/disk2/tengjianping/doris-master/be/src/vec/exprs/vexpr_context.cpp:98:5
    #2 0x55b2e757b6d8 in doris::vectorized::VScanner::prepare(doris::RuntimeState*, std::vector<std::shared_ptr<doris::vectorized::VExprContext>, std::allocator<std::shared_ptr<doris::vectorized::VExprContext>>> const&) /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/vscanner.cpp:47:13
    #3 0x55b2e78e8155 in doris::vectorized::NewOlapScanner::init() /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/new_olap_scanner.cpp:109:5
    #4 0x55b2e7551c81 in doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, std::shared_ptr<doris::vectorized::VScanner>) /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/scanner_scheduler.cpp:279:27
    #5 0x55b2e7554d5e in doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()::operator()() const /mnt/disk2/tengjianping/doris-master/be/src/vec/exec/scan/scanner_scheduler.cpp:202:31
    #6 0x55b2e7554c14 in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #7 0x55b2e7554bb4 in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&>(doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
    #8 0x55b2e7554a1c in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_0::operator()() const::'lambda0'()>::_M_invoke(std::_Any_data const&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
    #9 0x55b2c80f2cd2 in std::function<void ()>::operator()() const /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
    #10 0x55b2e755f3e4 in doris::PriorityWorkStealingThreadPool::work_thread(int) /mnt/disk2/tengjianping/doris-master/be/src/util/priority_work_stealing_thread_pool.hpp:135:17
    #11 0x55b2e7563c72 in void std::__invoke_impl<void, void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>(std::__invoke_memfun_deref, void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14
    #12 0x55b2e7563b44 in std::__invoke_result<void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>::type std::__invoke<void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>(void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    #13 0x55b2e7563b14 in decltype(std::__invoke((*this)._M_pmf, std::forward<doris::PriorityWorkStealingThreadPool*&>(fp), std::forward<int&>(fp))) std::_Mem_fn_base<void (doris::PriorityWorkStealingThreadPool::*)(int), true>::operator()<doris::PriorityWorkStealingThreadPool*&, int&>(doris::PriorityWorkStealingThreadPool*&, int&) const /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:131:11
    #14 0x55b2e7563ae4 in void std::__invoke_impl<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>(std::__invoke_other, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #15 0x55b2e7563a54 in std::enable_if<is_invocable_r_v<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>, void>::type std::__invoke_r<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>(std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
    #16 0x55b2e75639c3 in void std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>::__call<void, 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:570:11
    #17 0x55b2e756382d in void std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>::operator()<>() /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:629:17
    #18 0x55b2e7563744 in void std::__invoke_impl<void, std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>(std::__invoke_other, std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>&&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #19 0x55b2e7563704 in std::__invoke_result<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>::type std::__invoke<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>(std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>&&) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    #20 0x55b2e75636dc in void std:🧵:_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:253:13
    #21 0x55b2e75636b4 in std:🧵:_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>::operator()() /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:260:11
    #22 0x55b2e7563638 in std:🧵:_State_impl<std:🧵:_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>>::_M_run() /mnt/disk2/tengjianping/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13
    #23 0x55b2eb41d0ef in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18
    #24 0x7f1dfd4e1179 in start_thread pthread_create.c
    #25 0x7f1dfdd7bdf2 in clone (/lib64/libc.so.6+0xfcdf2) (BuildId: 20ee73ce1b6ac38a52440bab82ec7e28f0f5c5b9)
 2023-06-07 17:00:29 +08:00
bingquanzhao
7ead287405 [typo](doc)Fixed some date function documentation descriptions (#20539) 2023-06-07 16:30:19 +08:00
catpineapple
40205503d7 [doc](fix) update cold hot separation chinese doc (#20542) 2023-06-07 16:29:57 +08:00
zhangdong
c910e9b78b [doc](disk)fix disk capacity doc error (#20506) 2023-06-07 15:20:04 +08:00
jakevin
6b325a8458 [fix](Nereids): union output can be Alias<Slot> (#20532) 2023-06-07 15:11:07 +08:00
yuxuan-luo
732526ba58 [doc](multi-catalog) paimon catalog doc (#20503) 2023-06-07 14:21:07 +08:00
minghong
a4acd2c06e [chore](gitignore) Update gitignore to skip: tpcds tools and third party zip (#20529) 
* upate git ignore: tpcds tools, third party zip
 2023-06-07 13:02:10 +08:00
Pxl
36216f0925 [Bug](Agg-State) fix coredump when state combinator input const column (#20510) 
fix coredump when state combinator input const column
 2023-06-07 11:28:55 +08:00
starocean999
cd70c37402 [fix](nereids) filter and project node should be pushed down through cte (#20508) 
1.move PushdownFilterThroughCTEAnchor and PushdownProjectThroughCTEAnchor into PUSH_DOWN_FILTERS rule set
2.move PushdownFilterThroughProject before MergeProjectPostProcessor
 2023-06-07 10:36:32 +08:00
AKIRA
b83039de76 [fix](stats) Make alter column stats no forward (#20501) 
For test convenient, since daily regression tests queries would be sent any FE rather than master only.
 2023-06-07 10:14:44 +08:00
Jerry Hu
03a679e33a [improvement](sink) reuse rows buffer in msyql_result_writer (#20482) 
Creating a rows buffer for each block can impact non-negligible performance.
So it is necessary to reuse the rows buffer.

Test with a total of 1.7M rows, the AppendBatchTime reduced from 500ms to 280ms.
 2023-06-07 10:09:32 +08:00
zhangstar333
35a2be1074 [refactor](agg_state) refactor agg_state type to support fixed length object type (#20370) 
before the agg_state type only support with datatype string,
But with some agg functions, eg: avg,sum,mix...
those functions need serialize type is fixed length object type
 2023-06-07 10:05:00 +08:00
Super .Wein（星痕）
105d9a9513 [typo](doc) Fix a wrongly written in the release note for 1.2.0 version (#20496) 2023-06-07 09:39:42 +08:00
Jerry Hu
49f8f20fb1 [fix](regex) String with Chinese characters matching failed (#20493) 2023-06-07 07:27:47 +08:00
zhangstar333
3691372054 [bug](table_function) fix table function node forget to call open function of expr (#20495) 2023-06-07 07:26:50 +08:00
wangbo
3aa0c2bdbf [Bug](memleak) Fix emptyoperator may cause node not close (#20525) 2023-06-07 01:27:13 +08:00
yuxuan-luo
b65094c8df [Improvement](multi-catalog) paimon supports projection push down (#20522) 
Co-authored-by: hugoluo <hugoluo@tencent.com>
 2023-06-07 00:39:08 +08:00
gnehil
880e2d8373 [typo](doc) update spark connnector version compatibility instructions (#20477) 2023-06-06 23:33:27 +08:00
bingquanzhao
43ae2c59c3 [typo](doc) Fixed some description of date_format function documentation (#20504) 2023-06-06 23:32:34 +08:00
Hong Liu
5a749e6f4d [doc](catalog-hive) Add the property of hive catalog with kerberos. (#20502) 
Co-authored-by: smallhibiscus <844981280>
 2023-06-06 23:30:19 +08:00
AlexYue
c991249360 [enhancement](cooldown) use cooldown replica first when generating scan node (#20384) 2023-06-06 22:15:49 +08:00
AlexYue
a68afd0672 [fix](cooldown) fix bug due to tablets info changed (#20465) 2023-06-06 22:15:17 +08:00
zhannngchen
b22e364cdb [fix](log) publish version log is printed too frequently (#20507) 2023-06-06 20:34:38 +08:00
morrySnow
82cf76f92b [fix](Nereids) join condition not extract as conjunctions (#20498) 2023-06-06 20:34:19 +08:00
mch_ucchi
05bdbce8fc [Feature](Nereids) support update unique table statement (#20313) 2023-06-06 20:32:43 +08:00
zgxme
61d9bd2ba1 [fix](regression) fix export file test cases (#20463) 2023-06-06 20:07:31 +08:00
wudi
1f63c56e20 [sample](doris-soruce) add demo for reading data from doris be using thrift (#20192) 2023-06-06 19:57:34 +08:00
minghong
0c6292abaa [fix](stats) skip forbid_unknown_col_stats check for invisible column and internal db (#20362) 
1. skip forbidUnknownColStats check for in-visible columns
2. use columsStatistics.isUnknown to tell if this stats is unknown
3. skip unknown stats check for internal schema
 2023-06-06 19:07:33 +08:00
Xin Liao
625a8bcb05 [fix](merge-on-write) fix that set_txn_related_delete_bitmap may coredump (#20300) 2023-06-06 17:49:01 +08:00
Chengpeng Yan
a569d371b3 [fix](Nereids) give clean error message when there are subquery in the on clause (#20211) 
Add the rule for checking the join node in `analysis/CheckAnalysis.java` file. When we check the join node, we should check its' on clause. If there are some subquery expression, we should throw exception.

Before this PR
```
mysql> select a.k1 from baseall a join test b on b.k2 in (select 49);
ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: nul
```

After this PR
```
mysql> select a.k1 from baseall a join test b on b.k2 in (select 49);
ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: Not support OnClause contain Subquery, expr:k2 IN (INSUBQUERY) (LogicalOneRowRelation ( projects=[49 AS `49`#28], buildUnionNode=true ))
```
 2023-06-06 16:50:20 +08:00
Lijia Liu
b1a8bb28f7 [Fix](WorkloadGroup)Fix query queue nereids bug #20484 2023-06-06 16:44:35 +08:00
TengJianPing
4bc221aa25 [improvement](column reader) lazy load indices (#20456) 
Currently when reading column data, all types of indice are read even if they are not actually used, this PR implements lazy load of indices.
 2023-06-06 16:36:06 +08:00
lsy3993
17259672ff [typo](docs)modify http_port to webserver_port (#20447) 2023-06-06 16:08:45 +08:00
camby
7df8459e21 [fix](regression-test) add retry time to avoid regression test failed (#20487) 
Now after alter table ${tbl} set('dynamic_partition.end'='5'), we add dynamic partition async.
We need to wait dynamic scheduler.
 2023-06-06 15:50:11 +08:00
lvshaokang
48021366bf [fix](load) fix unified load redirect status delegate error (#20467) 2023-06-06 15:46:48 +08:00
yujun
13f1b90768 [Fix] (tablet) fix tablet queryable set (#20413) (#20414) 2023-06-06 15:38:01 +08:00
FreeOnePlus
24f9610cbb [fix](docker)Add container graceful exit logic (#20474) 
Add FE container and BE container to execute the logic of the Stop script when executing the exit command to ensure that the metadata is written successfully and minimize the restart exception caused by BEBJE.
 2023-06-06 15:25:21 +08:00
FreeOnePlus
1b02b28c40 [feature](docker)Docker example hive-broker-doris (#20473) 
add new docker example: hdfs-broker-doris
 2023-06-06 15:24:58 +08:00
FreeOnePlus
a3bcdf7b44 [docs](docs)Fix docker example doc (#20472) 
fix run-docker-cluster docs docker volumes:conf
 2023-06-06 15:24:32 +08:00
FreeOnePlus
0337dd573c [fix](docker)Fix docker example script (#20471) 
remove docker example script volumes: conf
 2023-06-06 15:24:04 +08:00
FreeOnePlus
f1db1f3663 [fix](docker)Fix BE init script Bug (#20470) 
change /bin/env bash -> /bin/bash
 2023-06-06 15:23:35 +08:00
FreeOnePlus
5184b31620 [feature](docker)Add new example MySQL-Flink-Doris Demo (#20469) 
Add new example MySQL-Flink-Doris Demo
 2023-06-06 15:23:11 +08:00
yuxuan-luo
fe63a0a3bb [Feature](multi-catalog)support paimon catalog (#19681) 
CREATE CATALOG paimon_n2 PROPERTIES (
"dfs.ha.namenodes.HDFS1006531" = "nn2,nn1",
"dfs.namenode.rpc-address.HDFS1006531.nn2" = "172.16.65.xx:4007",
"dfs.namenode.rpc-address.HDFS1006531.nn1" = "172.16.65.xx:4007",
"hive.metastore.uris" = "thrift://172.16.65.xx:7004",
"type" = "paimon",
"dfs.nameservices" = "HDFS1006531",
"hadoop.username" = "hadoop",
"paimon.catalog.type" = "hms",
"warehouse" = "hdfs://HDFS1006531/data/paimon1",
"dfs.client.failover.proxy.provider.HDFS1006531" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"
);
 2023-06-06 15:08:30 +08:00
Chengpeng Yan
ae428c29e2 [feature](planner)(nereids) support user defined variable (#20334) 
Support user-defined variables.
After this PR, we can use `set @a = xx` to define a user variable and use it in the query like `select @a`.

the changes of this PR:
1. Support the grammar for `set user variable` in the parser.
2. Add the `userVars` in `VariableMgr` to store the user-defined variables.
3. For the `set @a = xx`, we will store the variable name and its value in the `userVars` in `VariableMgr`.
4. For the `select @a`, we will get the value for the variable name in `userVars`.
 2023-06-06 14:35:16 +08:00
yuxuan-luo
0fce7b9011 [fix](http) Let the sdk find the httpclient package determined (#20205) 2023-06-06 14:20:38 +08:00