Commit Graph

8748 Commits

Author SHA1 Message Date
f58032f1da [fix](dynamic partition) drop partition exclude history_partition_num #37539 (#37570)
cherry pick from #37539

---------

Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
2024-07-10 22:09:00 +08:00
a7416f6ff9 [fix](dump) do not report success if dumping fails (#37510)
## Proposed changes
pick https://github.com/apache/doris/pull/37508

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-10 16:26:14 +08:00
87cd366636 [chore](dependencies)Remove Unnecessary Dependencies (#37469) (#37555)
bp #37469
2024-07-10 09:33:43 +08:00
770e7d21a4 [dependencies](fe)upgrade paimon to 0.8.1 (#37205) (#37554)
bp #37205
2024-07-10 09:32:33 +08:00
db4d061a68 [fix](Nereids) null type result with alias name should keep alias name (#37457) (#37524)
pick from master #37457
2024-07-09 20:46:51 +08:00
8ef83259ff [fix](planner) fix bug of select stmt toSql(#37274) (#37344)
cherry-pick from master #37274
2024-07-09 20:34:57 +08:00
9b075bc873 [fix](nereids) derive column stats for 'expr and A is not null' (#37235) (#37498)
pick from #37235 
the algorithm for computing stats for "expr1 and expr2" predicate is as
following:
1. compute output stats of expr1 based on input stats. the result stats
is denoted by leftStats
2. compute stats of expr2 based on leftStats after step1, leftStats
should be normalized to avoid abnormal cases, such as ndv > rowCount or
numNulls > rowCount

Issue Number: close #xxx

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-09 17:46:57 +08:00
7f6b846f58 [improve](fe) Add a counter metric for recording large editlog write (#37328) (#37474) 2024-07-09 17:16:31 +08:00
9b500faa0c [fix](create table) create table fail not write drop table editlog #37488 (#37506)
cherry pick from #37488
2024-07-09 13:44:43 +08:00
4426d6d80f [fix](fe) Add check editlog size mechanism for backupJob (#35653) (#37466)
* When creating a backupJob with huge of tables in a database, it can
cause backupJob editlog size over 2GB and bdbje will throw exception
because of ByteBuffer overflow

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-09 10:33:28 +08:00
ca0e44f83f [fix](case) fix struct format out files (#37350) (#37499)
bp #37350
2024-07-09 10:11:50 +08:00
a6ec78ec5f [fix](truncate) fix tablet invert index leaky #37334 (#37410)
cherry pick from #37334
2024-07-08 20:57:30 +08:00
4c5a7b26e2 [chore](restore) Log partition visible version (#37414)
Cherry-pick #36920.
2024-07-08 18:58:16 +08:00
68352c7c77 [fix](Nereids) constant folding for str_to_date on datev1 is wrong (#37360) (#37367)
pick from master #37360

when enable datev1, we should return datev1/datetimev1
2024-07-08 15:21:12 +08:00
fbc954e8be [feat](mtmv) Support grouping_sets rewrite when query rewrite by materialized view (#36056) (#37436)
cherry pick from master
pr: #36056
commitId: 569c9772
2024-07-08 15:06:16 +08:00
779a51570e [opt](mtmv) Set query rewrite by materialized view default enable (#35897) (#36949)
cherry pick from master
pr: #35897
commitId: 603fa82f
2024-07-08 14:29:38 +08:00
95dad14062 [chore](query) print query id when killed by timeout checker (#37402)
pick #36868
2024-07-08 11:26:29 +08:00
dd18652861 [branch-2.1](routine-load) make get Kafka meta timeout configurable (#37399)
pick #36619
2024-07-08 10:39:17 +08:00
af7b69da48 [fix](nereids) Pick the stop watch is not reset (#37168) (#37397)
Pick https://github.com/apache/doris/pull/37168 and
https://github.com/apache/doris/pull/37095
2024-07-08 10:28:03 +08:00
97e4025ee0 [branch-2.1](routine-load) increase routine load job default max batch size and rows (#37388)
pick #36632

Most users only care about the size of **max_batch_interval**, but in
order to achieve an interval effect, they have to configure
**max_batch_rows** and **max_batch_size** according to the
characteristics of the data. By adjusting these two default values,
users do not need to worry about configuration in most scenarios.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-07 18:35:08 +08:00
a05406ecc9 [branch-2.1] Picks "[Fix](delete) Fix delete job timeout when executing delete from ... #37363" (#37374)
## Proposed changes

picks https://github.com/apache/doris/pull/37363
2024-07-07 18:33:17 +08:00
d08a418dd8 [branch-2.1](routine-load) optimize routine load job auto resume policy (#37373)
pick #35266
2024-07-07 18:16:56 +08:00
af960f7c70 [branch-2.1](routine-load) dealing with the high watermark of Kafka may fallback (#37372)
pick #35901
2024-07-07 18:15:54 +08:00
960b02f293 [branch-2.1](routine-load) add retry when get Kafka meta info (#37371)
pick #35376
2024-07-07 18:14:38 +08:00
38b3870fe8 [branch-2.1] Picks "[fix](autoinc) Fix AutoIncrementGenerator and add more logs about auto-increment column #37306" (#37366)
## Proposed changes

picks https://github.com/apache/doris/pull/37306
2024-07-06 16:53:29 +08:00
7ce4a42d28 [fix](fe) Fix the sql of AddPartitionRecord (#37341)
Cherry-pick #37295

The range field is accidentally compared to DUMMY_ITEM.

It was introduced by #35461.
2024-07-05 22:05:12 +08:00
5de6aa74c0 [branch-2.1] Picks "[opt](autoinc) Forbid some schema change when the table has auto-increment column #37186" (#37331)
## Proposed changes

picks https://github.com/apache/doris/pull/37186
2024-07-05 21:59:30 +08:00
8a0d05d9b0 [opt](mtmv) Materialized view partition track supports date_trunc and optimize the fail reason (#35562) (#36947)
cherry pick from master #35562
commitId: 43d0f191
2024-07-05 15:12:43 +08:00
f8cee439b6 [feature](ES Catalog) map nested/object type in ES to JSON type in Doris (#37101) (#37182)
backport #37101
2024-07-05 10:48:32 +08:00
256221a574 [fix](Nereids) normalize aggregate should not push down lambda's param (#37109) (#37285)
pick from master #37109

ArrayItemSlot should not be inputSlot
2024-07-05 09:33:57 +08:00
8373610281 [opt](ctas) add a variable to control varchar length in ctas (#37069) (#37284)
pick from master #37069

add a new session variable: use_max_length_of_varchar_in_ctas

In CTAS (Create Table As Select), if CHAR/VARCHAR columns do not
originate from the source table, whether to set the length of such a
column to MAX, which is 65533. The default is true.
2024-07-04 22:09:41 +08:00
4e4f3d204e [feat](Nereids) push down predicates with multi columns through LogicalWindow and LogicalPartitionTopN (#36828) (#36981)
cherry-pick #36828 to branch-2.1

The requirement for predicate pushdown through the window operator is
that the partition by slots of the window contains all slots in the
predicate. The original implementation of doris only allows predicate
pushdown with one slot. This PR relaxes this restriction and allows for
predicate pushdown with multiple slots. The same applies to the
predicate pushdown of the LogicalPartitionTopN operator. The following
sql is an example.

select
	*
from
	(
	select
		row_number() over(partition by id, value1 order by value1) as num,
		id,
		value1
	from
		push_down_multi_column_predicate_through_window_t ) t
where
	abs(id + value1)<4
	and num <= 2;


Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2024-07-04 21:00:08 +08:00
6ec0476412 [chore](Nereids) opt part not exists error msg in bind relation (#36792)(#37160) (#37280)
pick from master #36792 #37160

print table name when partition not exists in bind relation
2024-07-04 19:19:36 +08:00
c7ad1f3d21 [fix](Nereids) simplify window expression should inherit data type (#37061) (#37283)
pick from master #37061

after window expression rewritten by literal.
literal's data type should same with original window expression.
2024-07-04 19:19:05 +08:00
e4fb506c20 [fix](Nereids) null type in result set will be cast to tinyint (#37019) (#37281)
pick from master #37019
2024-07-04 19:18:35 +08:00
ceef9ee123 [feature](serde) support presto compatible output format (#37039) (#37253)
bp #37039
2024-07-04 13:56:05 +08:00
3613413a54 [fix](hive) support find serde info from both tbl properties and serde properties (#37043) (#37188)
bp #37043
2024-07-04 13:55:38 +08:00
5f3e1e44b2 [bugfix]thread pool resource leak for 2.1 #36990 (#37247)
bp: #36990
2024-07-04 11:23:47 +08:00
26be313d40 [mv](nereids) mv cost related PRs (#35652 #35701 #35864 #36368 #36789 #34970) (#37097)
## Proposed changes
pick from #35652 #35701 #35864 #36368 #36789 #34970

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-04 09:42:11 +08:00
077fda4259 [enhance](mtmv)show create materialized view (#36188) (#37125)
pick: https://github.com/apache/doris/pull/36188
2024-07-03 22:48:43 +08:00
69aebc2d25 [branch-2.1] Picks "[Fix](schema change) Fix can't do reorder column schema change for MOW table and duplicate key table #37067" (#37226)
## Proposed changes

picks https://github.com/apache/doris/pull/37067
2024-07-03 22:42:51 +08:00
84f5bb73da [refactor](nereids) refactor analyze view (#37106) (#37163)
The Analyzer of NereidsPlanner use different rules to analyze normal plan and view, to prevent the plans in the views analyze multiple times, because some rules can not apply multiple times, say, decimal type coercion, if this rule applied multiple times, it will generate a wrong result.

But this design is trick. Normally, after process the LogicalView, the whole plan tree in the LogicalView should not contains unbound plan, but the current situation is not like this. And this problem block development of some rules, so I refactor it:
1. the Analyzer will not traverse the children of the LogicalView
2. After link the LogicalView to the outer plan tree, the whole plan tree of the LogicalView will not contains unbound plan
3. analyze view and table use the same rules, keep it simple
2024-07-03 19:09:49 +08:00
45fc1c7182 [opt](hive) save hive table schema in transaction for 2.1 (#37127)
## Proposed changes

pick #37008
2024-07-03 17:32:58 +08:00
b3f2bd20e3 [feat](nereids) support explain delete from clause #36782 (#37100)
## Proposed changes
pick from  #36782

support explain like:
explain delete from T where A=1

Issue Number: close #xxx

<!--Describe your changes.-->

(cherry picked from commit dc369cd13096dbb90700f7fbf8f35a9059d9906f)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-03 15:08:08 +08:00
5969d6521f [branch-2.1](function) fix nereids fold constant wrong result of abs (#37065) (#37108)
pick https://github.com/apache/doris/pull/37065
2024-07-03 11:58:06 +08:00
fb642d0227 [Fix](hive-writer) Fixed the issue where uncompletedMpuPendingUploads did not remove objects correctly. (#37173)
Backport #36905.
2024-07-03 11:09:46 +08:00
4761090848 [fix](tvf) Partition columns in CTAS need to be compatible with the STRING type of external tables/TVF (#37161)
bp: #35489
2024-07-03 10:58:08 +08:00
17056dcdb9 [Fix](multi-catalog) Fix the transaction is not removed in abnormal situations by removing transaction in finally block. (#37172)
Backport #36705.
2024-07-03 10:56:41 +08:00
0aeb768bf9 [Fix](export/outfile) Support compression when exporting data to Parquet / ORC. (#37167)
bp: #36490
2024-07-03 10:53:57 +08:00
6804ae5827 [enhance](mtmv)reduce the behavior of triggering the mtmv state to change to schema_change (#36513) (#37122)
pick from https://github.com/apache/doris/pull/36513
2024-07-03 10:42:03 +08:00