database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,291 Commits 1 Branch 0 Tags
7f6b846f5873ed0f76e43f21f2eda905ec5be6ca
Commit Graph

7215 Commits

Author SHA1 Message Date
Lei Zhang
7f6b846f58 [improve](fe) Add a counter metric for recording large editlog write (#37328) (#37474) 2024-07-09 17:16:31 +08:00
yujun
9b500faa0c [fix](create table) create table fail not write drop table editlog #37488 (#37506) 
cherry pick from #37488
 2024-07-09 13:44:43 +08:00
Lei Zhang
4426d6d80f [fix](fe) Add check editlog size mechanism for backupJob (#35653) (#37466) 
* When creating a backupJob with huge of tables in a database, it can
cause backupJob editlog size over 2GB and bdbje will throw exception
because of ByteBuffer overflow

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-07-09 10:33:28 +08:00
Mingyu Chen
ca0e44f83f [fix](case) fix struct format out files (#37350) (#37499) 
bp #37350
 2024-07-09 10:11:50 +08:00
yujun
a6ec78ec5f [fix](truncate) fix tablet invert index leaky #37334 (#37410) 
cherry pick from #37334
 2024-07-08 20:57:30 +08:00
walter
4c5a7b26e2 [chore](restore) Log partition visible version (#37414) 
Cherry-pick #36920.
 2024-07-08 18:58:16 +08:00
morrySnow
68352c7c77 [fix](Nereids) constant folding for str_to_date on datev1 is wrong (#37360) (#37367) 
pick from master #37360

when enable datev1, we should return datev1/datetimev1
 2024-07-08 15:21:12 +08:00
seawinde
fbc954e8be [feat](mtmv) Support grouping_sets rewrite when query rewrite by materialized view (#36056) (#37436) 
cherry pick from master
pr: #36056
commitId: 569c9772
 2024-07-08 15:06:16 +08:00
seawinde
779a51570e [opt](mtmv) Set query rewrite by materialized view default enable (#35897) (#36949) 
cherry pick from master
pr: #35897
commitId: 603fa82f
 2024-07-08 14:29:38 +08:00
hui lai
95dad14062 [chore](query) print query id when killed by timeout checker (#37402) 
pick #36868
 2024-07-08 11:26:29 +08:00
hui lai
dd18652861 [branch-2.1](routine-load) make get Kafka meta timeout configurable (#37399) 
pick #36619
 2024-07-08 10:39:17 +08:00
meiyi
af7b69da48 [fix](nereids) Pick the stop watch is not reset (#37168) (#37397) 
Pick https://github.com/apache/doris/pull/37168 and
https://github.com/apache/doris/pull/37095
 2024-07-08 10:28:03 +08:00
hui lai
97e4025ee0 [branch-2.1](routine-load) increase routine load job default max batch size and rows (#37388) 
pick #36632

Most users only care about the size of **max_batch_interval**, but in
order to achieve an interval effect, they have to configure
**max_batch_rows** and **max_batch_size** according to the
characteristics of the data. By adjusting these two default values,
users do not need to worry about configuration in most scenarios.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-07-07 18:35:08 +08:00
bobhan1
a05406ecc9 [branch-2.1] Picks "[Fix](delete) Fix delete job timeout when executing delete from ... #37363" (#37374) 
## Proposed changes

picks https://github.com/apache/doris/pull/37363
 2024-07-07 18:33:17 +08:00
hui lai
d08a418dd8 [branch-2.1](routine-load) optimize routine load job auto resume policy (#37373) 
pick #35266
 2024-07-07 18:16:56 +08:00
hui lai
af960f7c70 [branch-2.1](routine-load) dealing with the high watermark of Kafka may fallback (#37372) 
pick #35901
 2024-07-07 18:15:54 +08:00
hui lai
960b02f293 [branch-2.1](routine-load) add retry when get Kafka meta info (#37371) 
pick #35376
 2024-07-07 18:14:38 +08:00
bobhan1
38b3870fe8 [branch-2.1] Picks "[fix](autoinc) Fix AutoIncrementGenerator and add more logs about auto-increment column #37306" (#37366) 
## Proposed changes

picks https://github.com/apache/doris/pull/37306
 2024-07-06 16:53:29 +08:00
walter
7ce4a42d28 [fix](fe) Fix the sql of AddPartitionRecord (#37341) 
Cherry-pick #37295

The range field is accidentally compared to DUMMY_ITEM.

It was introduced by #35461.
 2024-07-05 22:05:12 +08:00
bobhan1
5de6aa74c0 [branch-2.1] Picks "[opt](autoinc) Forbid some schema change when the table has auto-increment column #37186" (#37331) 
## Proposed changes

picks https://github.com/apache/doris/pull/37186
 2024-07-05 21:59:30 +08:00
seawinde
8a0d05d9b0 [opt](mtmv) Materialized view partition track supports date_trunc and optimize the fail reason (#35562) (#36947) 
cherry pick from master #35562
commitId: 43d0f191
 2024-07-05 15:12:43 +08:00
qiye
f8cee439b6 [feature](ES Catalog) map nested/object type in ES to JSON type in Doris (#37101) (#37182) 
backport #37101
 2024-07-05 10:48:32 +08:00
morrySnow
256221a574 [fix](Nereids) normalize aggregate should not push down lambda's param (#37109) (#37285) 
pick from master #37109

ArrayItemSlot should not be inputSlot
 2024-07-05 09:33:57 +08:00
morrySnow
8373610281 [opt](ctas) add a variable to control varchar length in ctas (#37069) (#37284) 
pick from master #37069

add a new session variable: use_max_length_of_varchar_in_ctas

In CTAS (Create Table As Select), if CHAR/VARCHAR columns do not
originate from the source table, whether to set the length of such a
column to MAX, which is 65533. The default is true.
 2024-07-04 22:09:41 +08:00
feiniaofeiafei
4e4f3d204e [feat](Nereids) push down predicates with multi columns through LogicalWindow and LogicalPartitionTopN (#36828) (#36981) 
cherry-pick #36828 to branch-2.1

The requirement for predicate pushdown through the window operator is
that the partition by slots of the window contains all slots in the
predicate. The original implementation of doris only allows predicate
pushdown with one slot. This PR relaxes this restriction and allows for
predicate pushdown with multiple slots. The same applies to the
predicate pushdown of the LogicalPartitionTopN operator. The following
sql is an example.

select
	*
from
	(
	select
		row_number() over(partition by id, value1 order by value1) as num,
		id,
		value1
	from
		push_down_multi_column_predicate_through_window_t ) t
where
	abs(id + value1)<4
	and num <= 2;


Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
 2024-07-04 21:00:08 +08:00
morrySnow
6ec0476412 [chore](Nereids) opt part not exists error msg in bind relation (#36792)(#37160) (#37280) 
pick from master #36792 #37160

print table name when partition not exists in bind relation
 2024-07-04 19:19:36 +08:00
morrySnow
c7ad1f3d21 [fix](Nereids) simplify window expression should inherit data type (#37061) (#37283) 
pick from master #37061

after window expression rewritten by literal.
literal's data type should same with original window expression.
 2024-07-04 19:19:05 +08:00
morrySnow
e4fb506c20 [fix](Nereids) null type in result set will be cast to tinyint (#37019) (#37281) 
pick from master #37019
 2024-07-04 19:18:35 +08:00
Mingyu Chen
ceef9ee123 [feature](serde) support presto compatible output format (#37039) (#37253) 
bp #37039
 2024-07-04 13:56:05 +08:00
Mingyu Chen
3613413a54 [fix](hive) support find serde info from both tbl properties and serde properties (#37043) (#37188) 
bp #37043
 2024-07-04 13:55:38 +08:00
wuwenchi
5f3e1e44b2 [bugfix]thread pool resource leak for 2.1 #36990 (#37247) 
bp: #36990
 2024-07-04 11:23:47 +08:00
minghong
26be313d40 [mv](nereids) mv cost related PRs (#35652 #35701 #35864 #36368 #36789 #34970) (#37097) 
## Proposed changes
pick from #35652 #35701 #35864 #36368 #36789 #34970

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-07-04 09:42:11 +08:00
zhangdong
077fda4259 [enhance](mtmv)show create materialized view (#36188) (#37125) 
pick: https://github.com/apache/doris/pull/36188
 2024-07-03 22:48:43 +08:00
bobhan1
69aebc2d25 [branch-2.1] Picks "[Fix](schema change) Fix can't do reorder column schema change for MOW table and duplicate key table #37067" (#37226) 
## Proposed changes

picks https://github.com/apache/doris/pull/37067
 2024-07-03 22:42:51 +08:00
924060929
84f5bb73da [refactor](nereids) refactor analyze view (#37106) (#37163) 
The Analyzer of NereidsPlanner use different rules to analyze normal plan and view, to prevent the plans in the views analyze multiple times, because some rules can not apply multiple times, say, decimal type coercion, if this rule applied multiple times, it will generate a wrong result.

But this design is trick. Normally, after process the LogicalView, the whole plan tree in the LogicalView should not contains unbound plan, but the current situation is not like this. And this problem block development of some rules, so I refactor it:
1. the Analyzer will not traverse the children of the LogicalView
2. After link the LogicalView to the outer plan tree, the whole plan tree of the LogicalView will not contains unbound plan
3. analyze view and table use the same rules, keep it simple
 2024-07-03 19:09:49 +08:00
wuwenchi
45fc1c7182 [opt](hive) save hive table schema in transaction for 2.1 (#37127) 
## Proposed changes

pick #37008
 2024-07-03 17:32:58 +08:00
minghong
b3f2bd20e3 [feat](nereids) support explain delete from clause #36782 (#37100) 
## Proposed changes
pick from  #36782

support explain like:
explain delete from T where A=1

Issue Number: close #xxx

<!--Describe your changes.-->

(cherry picked from commit dc369cd13096dbb90700f7fbf8f35a9059d9906f)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-07-03 15:08:08 +08:00
zclllyybb
5969d6521f [branch-2.1](function) fix nereids fold constant wrong result of abs (#37065) (#37108) 
pick https://github.com/apache/doris/pull/37065
 2024-07-03 11:58:06 +08:00
Qi Chen
fb642d0227 [Fix](hive-writer) Fixed the issue where uncompletedMpuPendingUploads did not remove objects correctly. (#37173) 
Backport #36905.
 2024-07-03 11:09:46 +08:00
Tiewei Fang
4761090848 [fix](tvf) Partition columns in CTAS need to be compatible with the STRING type of external tables/TVF (#37161) 
bp: #35489
 2024-07-03 10:58:08 +08:00
Qi Chen
17056dcdb9 [Fix](multi-catalog) Fix the transaction is not removed in abnormal situations by removing transaction in finally block. (#37172) 
Backport #36705.
 2024-07-03 10:56:41 +08:00
Tiewei Fang
0aeb768bf9 [Fix](export/outfile) Support compression when exporting data to Parquet / ORC. (#37167) 
bp: #36490
 2024-07-03 10:53:57 +08:00
zhangdong
6804ae5827 [enhance](mtmv)reduce the behavior of triggering the mtmv state to change to schema_change (#36513) (#37122) 
pick from https://github.com/apache/doris/pull/36513
 2024-07-03 10:42:03 +08:00
zhangdong
03942f23f6 [fix](mtmv)fix when related table drop partition,mv partition is sync… (#36602) 
pick: https://github.com/apache/doris/pull/36547
 2024-07-02 23:08:58 +08:00
zhangdong
5bb6642b86 [fix](mtmv)mtmv support default key (#36221) (#36601) 
pick : https://github.com/apache/doris/pull/36221
 2024-07-02 23:08:38 +08:00
zhangdong
fbca3196c5 [enhance](mtmv) not allow modify data of MTMV (#35870) (#37129) 
pick: https://github.com/apache/doris/pull/35870
 2024-07-02 23:06:45 +08:00
zhangdong
0a1abf10d6 [fix](auth)support check priv when tvf use resource (#36928) (#37132) 
pick https://github.com/apache/doris/pull/36928
 2024-07-02 23:03:29 +08:00
zhangdong
177764647d [fix](mtmv)fix mtmv dead lock (#37009) (#37133) 
pick https://github.com/apache/doris/pull/37009
 2024-07-02 23:01:31 +08:00
zhangdong
6716f9e736 [fix](mtmv)fix MTMVTask log has null pointer (#36840) (#37131) 
pick https://github.com/apache/doris/pull/36840
 2024-07-02 23:01:15 +08:00
zhangdong
65375b48fc [enhance](mtmv)support replace materialized view (#36749) (#37147) 
pick: https://github.com/apache/doris/pull/36749
 2024-07-02 23:00:55 +08:00