database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,239 Commits 1 Branch 0 Tags
4e4f3d204ecf30166e177a02ef38ec3e007e7dfd
Commit Graph

19239 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
feiniaofeiafei
4e4f3d204e [feat](Nereids) push down predicates with multi columns through LogicalWindow and LogicalPartitionTopN (#36828) (#36981) 
cherry-pick #36828 to branch-2.1

The requirement for predicate pushdown through the window operator is
that the partition by slots of the window contains all slots in the
predicate. The original implementation of doris only allows predicate
pushdown with one slot. This PR relaxes this restriction and allows for
predicate pushdown with multiple slots. The same applies to the
predicate pushdown of the LogicalPartitionTopN operator. The following
sql is an example.

select
	*
from
	(
	select
		row_number() over(partition by id, value1 order by value1) as num,
		id,
		value1
	from
		push_down_multi_column_predicate_through_window_t ) t
where
	abs(id + value1)<4
	and num <= 2;


Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
 2024-07-04 21:00:08 +08:00
daidai
c8978fc9d1 [fix](HadoopLz4BlockCompression)Fixed the bug that HadoopLz4BlockCompression creates _decompressor every time it decompresses.(#37187) (#37299) 
bp : #37187
 2024-07-04 20:22:27 +08:00
Mingyu Chen
55636e8035 [test](migrate) move 3 cases from p2 to p0 (#36957) (#37264) 
bp #36957

Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
 2024-07-04 20:09:59 +08:00
Pxl
e2c2702dff [Bug](runtime-filter) fix some rf error problems (#37155) 
## Proposed changes
pick from #37273
 2024-07-04 20:03:46 +08:00
morrySnow
6ec0476412 [chore](Nereids) opt part not exists error msg in bind relation (#36792)(#37160) (#37280) 
pick from master #36792 #37160

print table name when partition not exists in bind relation
 2024-07-04 19:19:36 +08:00
morrySnow
c7ad1f3d21 [fix](Nereids) simplify window expression should inherit data type (#37061) (#37283) 
pick from master #37061

after window expression rewritten by literal.
literal's data type should same with original window expression.
 2024-07-04 19:19:05 +08:00
morrySnow
e4fb506c20 [fix](Nereids) null type in result set will be cast to tinyint (#37019) (#37281) 
pick from master #37019
 2024-07-04 19:18:35 +08:00
wangbo
b272247a57 [pick]log thread num (#37258) 
## Proposed changes

pick #37159
 2024-07-04 15:27:52 +08:00
Mingyu Chen
ceef9ee123 [feature](serde) support presto compatible output format (#37039) (#37253) 
bp #37039
 2024-07-04 13:56:05 +08:00
Mingyu Chen
3613413a54 [fix](hive) support find serde info from both tbl properties and serde properties (#37043) (#37188) 
bp #37043
 2024-07-04 13:55:38 +08:00
wuwenchi
5f3e1e44b2 [bugfix]thread pool resource leak for 2.1 #36990 (#37247) 
bp: #36990
 2024-07-04 11:23:47 +08:00
Jibing-Li
bf3ea1839c [test]Mv external p2 test case to p0. (#37070) (#37140) 
backport: https://github.com/apache/doris/pull/37070
 2024-07-04 11:19:31 +08:00
TengJianPing
fb344b66ca [fix](hash join) fix numeric overflow when calculating hash table bucket size #37193 (#37213) 
## Proposed changes

Bp #37193
 2024-07-04 11:12:52 +08:00
Gabriel
4532ba990a [fix](pipeline) Avoid to close task twice (#36747) (#37115) 2024-07-04 10:02:56 +08:00
minghong
26be313d40 [mv](nereids) mv cost related PRs (#35652 #35701 #35864 #36368 #36789 #34970) (#37097) 
## Proposed changes
pick from #35652 #35701 #35864 #36368 #36789 #34970

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-07-04 09:42:11 +08:00
zhangdong
077fda4259 [enhance](mtmv)show create materialized view (#36188) (#37125) 
pick: https://github.com/apache/doris/pull/36188
 2024-07-03 22:48:43 +08:00
bobhan1
69aebc2d25 [branch-2.1] Picks "[Fix](schema change) Fix can't do reorder column schema change for MOW table and duplicate key table #37067" (#37226) 
## Proposed changes

picks https://github.com/apache/doris/pull/37067
 2024-07-03 22:42:51 +08:00
Pxl
70e1c563b3 [Chore](runtime-filter) enlarge sync filter size rpc timeout limit (#37103) (#37225) 
pick from #37103
 2024-07-03 21:02:26 +08:00
Pxl
ffc57c9ef4 [Bug](runtime-filter) fix brpc ctrl use after free (#37223) 
part of #35186
 2024-07-03 21:01:50 +08:00
zhannngchen
97945af947 [fix](merge-on-write) when full clone failed, duplicate key might occur (#37001) (#37229) 
cherry-pick #37001
 2024-07-03 19:48:10 +08:00
zy-kkk
a9f9113c48 [branch-2.1][test](external)move hive cases from p2 to p0 (#37149) 
pk (#36855)
test_hive_same_db_table_name
test_hive_special_char_partition
test_complex_types
test_wide_table
 2024-07-03 19:44:52 +08:00
924060929
84f5bb73da [refactor](nereids) refactor analyze view (#37106) (#37163) 
The Analyzer of NereidsPlanner use different rules to analyze normal plan and view, to prevent the plans in the views analyze multiple times, because some rules can not apply multiple times, say, decimal type coercion, if this rule applied multiple times, it will generate a wrong result.

But this design is trick. Normally, after process the LogicalView, the whole plan tree in the LogicalView should not contains unbound plan, but the current situation is not like this. And this problem block development of some rules, so I refactor it:
1. the Analyzer will not traverse the children of the LogicalView
2. After link the LogicalView to the outer plan tree, the whole plan tree of the LogicalView will not contains unbound plan
3. analyze view and table use the same rules, keep it simple
 2024-07-03 19:09:49 +08:00
wuwenchi
45fc1c7182 [opt](hive) save hive table schema in transaction for 2.1 (#37127) 
## Proposed changes

pick #37008
 2024-07-03 17:32:58 +08:00
Mingyu Chen
e5695e058f [test](migrate) move 2 cases from p2 to p0 (#36935) (#37200) 
bp #36935

Co-authored-by: zhangdong <493738387@qq.com>
 2024-07-03 17:29:01 +08:00
minghong
b3f2bd20e3 [feat](nereids) support explain delete from clause #36782 (#37100) 
## Proposed changes
pick from  #36782

support explain like:
explain delete from T where A=1

Issue Number: close #xxx

<!--Describe your changes.-->

(cherry picked from commit dc369cd13096dbb90700f7fbf8f35a9059d9906f)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-07-03 15:08:08 +08:00
zclllyybb
5969d6521f [branch-2.1](function) fix nereids fold constant wrong result of abs (#37065) (#37108) 
pick https://github.com/apache/doris/pull/37065
 2024-07-03 11:58:06 +08:00
Qi Chen
fb642d0227 [Fix](hive-writer) Fixed the issue where uncompletedMpuPendingUploads did not remove objects correctly. (#37173) 
Backport #36905.
 2024-07-03 11:09:46 +08:00
Qi Chen
e857680661 [Migrate-Test](multi-catalog) Migrate p2 tests from p2 to p0. (#37175) 
Backport #36989.
 2024-07-03 11:08:49 +08:00
Tiewei Fang
4761090848 [fix](tvf) Partition columns in CTAS need to be compatible with the STRING type of external tables/TVF (#37161) 
bp: #35489
 2024-07-03 10:58:08 +08:00
Qi Chen
17056dcdb9 [Fix](multi-catalog) Fix the transaction is not removed in abnormal situations by removing transaction in finally block. (#37172) 
Backport #36705.
 2024-07-03 10:56:41 +08:00
Tiewei Fang
0aeb768bf9 [Fix](export/outfile) Support compression when exporting data to Parquet / ORC. (#37167) 
bp: #36490
 2024-07-03 10:53:57 +08:00
zhangdong
6804ae5827 [enhance](mtmv)reduce the behavior of triggering the mtmv state to change to schema_change (#36513) (#37122) 
pick from https://github.com/apache/doris/pull/36513
 2024-07-03 10:42:03 +08:00
zhangdong
03942f23f6 [fix](mtmv)fix when related table drop partition,mv partition is sync… (#36602) 
pick: https://github.com/apache/doris/pull/36547
 2024-07-02 23:08:58 +08:00
zhangdong
5bb6642b86 [fix](mtmv)mtmv support default key (#36221) (#36601) 
pick : https://github.com/apache/doris/pull/36221
 2024-07-02 23:08:38 +08:00
zhangdong
fbca3196c5 [enhance](mtmv) not allow modify data of MTMV (#35870) (#37129) 
pick: https://github.com/apache/doris/pull/35870
 2024-07-02 23:06:45 +08:00
zhangdong
0a1abf10d6 [fix](auth)support check priv when tvf use resource (#36928) (#37132) 
pick https://github.com/apache/doris/pull/36928
 2024-07-02 23:03:29 +08:00
zhangdong
177764647d [fix](mtmv)fix mtmv dead lock (#37009) (#37133) 
pick https://github.com/apache/doris/pull/37009
 2024-07-02 23:01:31 +08:00
zhangdong
6716f9e736 [fix](mtmv)fix MTMVTask log has null pointer (#36840) (#37131) 
pick https://github.com/apache/doris/pull/36840
 2024-07-02 23:01:15 +08:00
zhangdong
65375b48fc [enhance](mtmv)support replace materialized view (#36749) (#37147) 
pick: https://github.com/apache/doris/pull/36749
 2024-07-02 23:00:55 +08:00
wuwenchi
b3eaf0e4d2 [bugfix](hive)Prevent multiple fs from being generated for 2.1 (#37142) 
pick #36954
 2024-07-02 22:54:40 +08:00
wuwenchi
64a79cc03a [bugfix](testcase)add java error log output for 2.1 (#37128) 
pick #35998
 2024-07-02 22:53:38 +08:00
wuwenchi
e7e1e967cf [test](migrate) move 2 cases from p2 to p0 for 2.1 (#37139) 
pick #37004
 2024-07-02 22:50:53 +08:00
Tiewei Fang
b445c783eb [test](tvf) move p2 tvf tests from p2 to p0 (#37081) (#37152) 
bp: #37081
 2024-07-02 22:38:22 +08:00
Tiewei Fang
74086189d3 [test](tvf) move p2 tvf tests from p2 to p0 (#36871) (#37150) 
bp: #36871
 2024-07-02 22:37:43 +08:00
Tiewei Fang
bd24a8bdd9 [Fix](csv_reader) Add a session variable to control whether empty rows in CSV files are read as NULL values (#37153) 
bp: #36668
 2024-07-02 22:12:17 +08:00
Mingyu Chen
e25717458e [opt](catalog) add some profile for parquet reader and change meta cache config (#37040) (#37146) 
bp #37040
 2024-07-02 20:58:43 +08:00
LiBinfeng
c7549acf29 [Fix](Nereids) fix leading with different be instance number (#36613) (#36967) 
cherry-pick: #36613

Problem:
When use different be number to test leading explain shape plan,
physical distribute plan would differ due to different be numbers
Solved:
Disable physical distribute node showing in fix leading cases
 2024-07-02 19:56:17 +08:00
924060929
42f4271e9d [enhancement](nereids) speedup sql cache with variable (#37090) (#37119) 
cherry pick from #37090
 2024-07-02 19:25:22 +08:00
wangbo
f5572ac732 [pick]reset memtable flush thread num (#37092) 
## Proposed changes

pick #37028
 2024-07-02 19:20:17 +08:00
qiye
b63e6ac350 [fix](ES Catalog)Add array types support in esquery function (#36936) (#37054) 
backport #36936
 2024-07-02 17:48:35 +08:00