database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
11,294 Commits 1 Branch 0 Tags
bf197ee8d2d9e6ccf88184663e4614068bc3edde
Commit Graph

11294 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
minghong
bf197ee8d2 [opt](nereids) adjust cost model for BroadCastJoin and PartitionJoin (#20713) 
we add penalty for broadcast join (bc for brief in the following).
the intuition of penalty is as follow:
1. if the build side is very small (< 1M), we prefer bc, and set `penalty=1`, which means no penalty
2. if build side is more than 1M, we consider the ratio of the probe row count to the build row count. the less the ratio is, the higher penalty is.

this pr has positive impact on tpch queries. Only q3 is changed. in out test (tpch 1T, 3BE) q3 improved from 5.1sec to 2.5 sec.
this pr has positive impact on tpcds queries. test on tpcds sf100 (3BE), cold run improve from 163 sec to 156 sec, hot run improves from 155 sec to 149 sec
 2023-06-16 22:49:04 +08:00
morrySnow
bb4f10b457 [fix](Nereids) lost having when analyze sort-having-agg (#20914) 2023-06-16 22:13:28 +08:00
morrySnow
7ee744ff5a [opt](Nereids) add more unexpected expression check (#20901) 
1. check sub-query after rewrite, should not meet any sub-query there
2. check below expression on specific plan
  - Aggreagate
    - TableGeneratingFunction
  - Filter
    - AggregateFunction
    - GroupingScalarFunction
    - TableGeneratingFunction
    - WindowExpression
  - Generate
    - AggregateFunction
    - GroupingScalarFunction
    - WindowExpression
  - Having
    - TableGeneratingFunction
    - WindowExpression
  - Join
    - AggregateFunction
    - GroupingScalarFunction
    - TableGeneratingFunction
    - WindowExpression
  - Project
    - TableGeneratingFunction
  - Sort
    - AggregateFunction
    - GroupingScalarFunction
    - TableGeneratingFunction
    - WindowExpression
  - Window
    - GroupingScalarFunction
    - TableGeneratingFunction
 2023-06-16 22:12:39 +08:00
nanfeng
483d96368c [doc](insert-overwrite) add related doc (#20657) 2023-06-16 21:11:04 +08:00
xzj7019
ab32299ba4 [feature](nereids) Support multi target rf #20714 
Support multi target runtime filter, mainly for set operation, such as union/intersect/except.
 2023-06-16 20:26:00 +08:00
YueW
1cc611a913 [fix](match) fix regression case test_index_match_select and test_index_match_phrase (#20860) 
1. add more checks for match expression in nereids:
  - match expression only support in filter
  - match expression left child and right child must all be string type
  - left child for match expression must be sloftRef, right child for match expression must be Literal

2. to fix regression case test_index_match_select and test_index_match_phrase
 2023-06-16 20:18:29 +08:00
morrySnow
5dc0f90c7f [opt](Nereids) revert convert IN with 2 options to OR expression rule (#20894) 
revert this rule because it has negative effect on predicate push-down-to-storage-layer
 2023-06-16 19:11:37 +08:00
jakevin
6cde7bc8ad [feature](Nereids) just reserve logical expression in memo after do dphyp (#20843) 
After DPHyp, clear all physicalExpression and other,
just keep logicalExpression as original plan as input of cascades optimize.
 2023-06-16 18:12:39 +08:00
zhannngchen
baed26acc6 [fix](merge-on-write) fix publish timeout (#20891) 2023-06-16 18:11:41 +08:00
starocean999
536bf56a35 [fix](planner) strip trailing zeros when cast string literal to decimal literal (#20743) 2023-06-16 17:08:36 +08:00
yongjinhou
2e295a1ee9 [Enhancement](http) unify http auth config (#20864) 2023-06-16 16:55:46 +08:00
minghong
22aee21208 [fix](nereids)disable tpcds query59 plan shape check #20886 
ds59 plan shape is not stable.
we will fix it in future
 2023-06-16 16:50:40 +08:00
AKIRA
e63739e729 [fix](nereids) add regression test for stats analyze and fix some bugs (#20865) 
1. Add regression test case for analyze to make sure show/drop/analyze stats would work as expected
2. Remove useless code, which would block the clean for expired stats
3. Fix bug of DropStats,  before this PR drop the whole table stats would casuse a NPE exception when parsing stmt
 2023-06-16 16:43:49 +08:00
mch_ucchi
0bd9aecfcc [Fix](planner&Nereids) fold constant invalid yyyy-mm-31 to the last day of month incorrectly (#20685) 
currently, expression: cast('20230631' as date) will be evaluate to 2023-06-30 incorrectly, and '20230632' will be null, we fix the bug and evaluate all the invalid date to null.
 2023-06-16 16:23:02 +08:00
mch_ucchi
5573858cb4 [Enhance](regression-test) use another db than test_query_db for nereids_p0 (#19467) 
replace test_query_db to nereids_test_query_db in nereids_p0 directory
split test_join into five files to run faster.
 2023-06-16 16:11:14 +08:00
amory
c3b9e99350 [fix](regress-test)update config for disable_nested_complex_type (#20735) 2023-06-16 15:51:41 +08:00
yuxuan-luo
97135a1cbb [Feature] (json)add json_contains function (#20824) 2023-06-16 15:10:12 +08:00
starocean999
ccfd6f1d23 [fix](nereids) only setHasColocatePlanNode when currentFragment's data partition is not RANDOM (#20885) 2023-06-16 14:54:22 +08:00
mch_ucchi
f741129ffe [Fix](regression-test) fix incorrect explain requirement when globally enable nereids (#20809) 2023-06-16 14:51:22 +08:00
chunping
731ce5802e [Test][Framework] add enableCacheData option for test framework (#20874) 2023-06-16 14:12:21 +08:00
Yongqiang YANG
1a62c79970 [fix](replica) do not delete the only one replica (#20872) 2023-06-16 13:57:18 +08:00
Chenyang Sun
f80bf3c1bc [fix](compaction) fix time series compaction policy (#20837) 2023-06-16 13:25:55 +08:00
Qi Chen
b7a50a09fe [Opt](orc-reader) Optimize orc reader by dict filtering. (#20806) 
Optimize orc reader by dict filtering.  It is similar with #17594.
Test result
**ssb-flat-100**: (3 nodes)
| Query        | before opt           | after opt  |
| ------------- |:-------------:| ---------:|
Q1.1 | 1.239 | 1.145
Q1.2 | 1.254 | 1.128
Q1.3 | 1.931 | 1.644
Q2.1 | 1.359 | 1.006
Q2.2 | 1.229 | 0.674
Q2.3 | 0.934 | 0.427
Q3.1 | 2.226 | 1.712
Q3.2 | 2.042 | 1.562
Q3.3 | 1.631 | 1.021
Q3.4 | 1.618 | 0.732
Q4.1 | 2.294 | 1.858
Q4.2 | 2.511 | 1.961
Q4.3 | 1.736 | 1.446
total | 22.004 | 16.316
 2023-06-16 13:11:37 +08:00
Qi Chen
722839e118 [Fix](multi-catalog) Fix hive transaction table regression test by adding hive-docker missing configurations. (#20832) 
Fix hive transaction table regression test test_transactional_hive by adding hive-docker missing configurations of #20679. Hive need to be set these configurations to do compaction.
 2023-06-16 13:08:24 +08:00
Xinyi Zou
1ec7f0e50a [fix](memory) memory management thread exits gracefully 
memory_maintenance_thread, memory_gc_thread, load_channel_tracker_refresh_thread, memory_tracker_profile_refresh_thread
 2023-06-16 11:40:24 +08:00
yagagagaga
05e5ebfdeb [typo][docs] Fix a parameter error (#20841) 2023-06-16 11:37:41 +08:00
Jack Drogon
9d41edd9eb [Feature](binlog) Add binlog gc && Auth master_token (#20854) 
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
 2023-06-16 11:25:11 +08:00
Adonis Ling
a52fc3eb33 [chore](workflow) Improve the cache hit ratio in BE UT (Clang) (#20871) 
In workflow BE UT (Clang), we set up the ldb_toolchain before we build the UT. During generating the toolchain tools, the script modifies the RPATH and dynamic linker of the executables which making the mtime of compilers change in every build.

By default, Ccache computes the hash of the compilers' mtime and size to check the compilers whether they are consistent with the ones which were used to generate the caches. If the compilers change, the caches can not be hit. We should change the default behavior of Ccache by setting the configuration `compiler_check (CCACHE_COMPILERCHECK)` to increase the cache hit ratio in the workflow.

Reference: https://ccache.dev/manual/latest.html#_configuration_options
 2023-06-16 11:18:21 +08:00
Pxl
db1da6b787 [Chore](pipeline) add some profile log when pipeline canceled (#20825) 
add some profile log when pipeline canceled
 2023-06-16 10:54:54 +08:00
minghong
7f413f13c4 [fix](nereids)use "analyze table XXX with sync" in regression cases #20856 
replace analyze table XXX by analyze table XXX with sync in regression test cases
 2023-06-16 10:39:37 +08:00
YueW
420603317b [improve](match) Improve performance for match query without inverted index (#20815) 2023-06-16 10:35:38 +08:00
Mryange
179b933ccc [test](regression) update some case in p2 (#20683) 2023-06-16 10:09:22 +08:00
yagagagaga
c3e6db827c [typo][docs] remove unuse config mysql_service_nio_enabled (#20862) 2023-06-16 09:58:33 +08:00
DongLiang-0
40050a46ca [fix](java-udf)Fix cannot download pentaho-aggdesigner-algorithm jar in java udf and format it (#20013) 2023-06-16 09:40:48 +08:00
bingquanzhao
9daa277b0d [typo](doc) Fix the description of the alter-table-partition document (#20800) 2023-06-15 21:27:59 +08:00
gnehil
71aba93e19 [typo](doc) spark connector add new param and a way to write bitmap and hll col (#20850) 2023-06-15 21:27:37 +08:00
caoliang-web
835a1e8af9 [typo](doc)delete document optimization (#20748) 2023-06-15 21:22:43 +08:00
bingquanzhao
6d4c16b497 [typo](doc) Fix formatting errors in CREATE-TABLE documents due to line indentation (#20839) 2023-06-15 21:22:01 +08:00
Jack Drogon
b38417358b [chore](compile) Remove fs/benchmark/fs_benchmark_tool.cpp in IO_FILES (#20845) 2023-06-15 21:16:59 +08:00
Xin Liao
f1af09ef87 [Enhancement](merge-on-write) parallel calculate delete bitmap when tablet has multi segments (#20706) 2023-06-15 21:11:39 +08:00
zy-kkk
e4e4f11b14 [typo](docs) Optimize the JDBC Catalog document (#20861) 2023-06-15 21:09:22 +08:00
jakevin
21bb7fea07 [minor](Nereids): convert if condition into a check (#20855) 2023-06-15 20:10:56 +08:00
Pxl
4bfceb7acb [Bug](pipeline) fix pipeline task call finish_p_dependency more than once (#20851) 
fix pipeline task call finish_p_dependency more than once

When pipeline task meet eos->PENDING_FINISH->CANCELED, this task will call finish_p_dependency twice.
 2023-06-15 19:21:40 +08:00
yuanyuan8983
11e164e7a5 [typo](doc) Added a description of inconsistent results (#20849) 2023-06-15 17:17:24 +08:00
HHoflittlefish777
bb5d36c5cb [Log](load) change VLOG to INFO when write replica failing (#20783) 2023-06-15 16:11:14 +08:00
Calvin Kirs
37f8334811 [Bug](GA)fix the wrong job dependencies (#20846) 2023-06-15 15:57:57 +08:00
Pxl
b6835840f7 [Bug](table-function) return InvalidArgument when explode_split meet empty delimiter (#20795) 
return InvalidArgument when explode_split meet empty delimiter
 2023-06-15 15:17:22 +08:00
zy-kkk
d9b3c2aba2 [improvement](jdbc) support support get mysql information_schema's table and clickhouse system's table (#20768) 2023-06-15 14:53:51 +08:00
Pxl
01e53f4e67 [Bug](materialized-view) fix problems about create mv on ssb_flat q4.1 failed (#20658) 
fix problems about create mv on ssb_flat q4.1 failed
 2023-06-15 14:38:21 +08:00
zgxme
e031ead53a [test](regression) support ccr regression (#20818) 
add CI base conf to support ccr regression
 2023-06-15 14:25:09 +08:00