Commit Graph

8717 Commits

Author SHA1 Message Date
4e4f3d204e [feat](Nereids) push down predicates with multi columns through LogicalWindow and LogicalPartitionTopN (#36828) (#36981)
cherry-pick #36828 to branch-2.1

The requirement for predicate pushdown through the window operator is
that the partition by slots of the window contains all slots in the
predicate. The original implementation of doris only allows predicate
pushdown with one slot. This PR relaxes this restriction and allows for
predicate pushdown with multiple slots. The same applies to the
predicate pushdown of the LogicalPartitionTopN operator. The following
sql is an example.

select
	*
from
	(
	select
		row_number() over(partition by id, value1 order by value1) as num,
		id,
		value1
	from
		push_down_multi_column_predicate_through_window_t ) t
where
	abs(id + value1)<4
	and num <= 2;


Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2024-07-04 21:00:08 +08:00
6ec0476412 [chore](Nereids) opt part not exists error msg in bind relation (#36792)(#37160) (#37280)
pick from master #36792 #37160

print table name when partition not exists in bind relation
2024-07-04 19:19:36 +08:00
c7ad1f3d21 [fix](Nereids) simplify window expression should inherit data type (#37061) (#37283)
pick from master #37061

after window expression rewritten by literal.
literal's data type should same with original window expression.
2024-07-04 19:19:05 +08:00
e4fb506c20 [fix](Nereids) null type in result set will be cast to tinyint (#37019) (#37281)
pick from master #37019
2024-07-04 19:18:35 +08:00
ceef9ee123 [feature](serde) support presto compatible output format (#37039) (#37253)
bp #37039
2024-07-04 13:56:05 +08:00
3613413a54 [fix](hive) support find serde info from both tbl properties and serde properties (#37043) (#37188)
bp #37043
2024-07-04 13:55:38 +08:00
5f3e1e44b2 [bugfix]thread pool resource leak for 2.1 #36990 (#37247)
bp: #36990
2024-07-04 11:23:47 +08:00
26be313d40 [mv](nereids) mv cost related PRs (#35652 #35701 #35864 #36368 #36789 #34970) (#37097)
## Proposed changes
pick from #35652 #35701 #35864 #36368 #36789 #34970

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-04 09:42:11 +08:00
077fda4259 [enhance](mtmv)show create materialized view (#36188) (#37125)
pick: https://github.com/apache/doris/pull/36188
2024-07-03 22:48:43 +08:00
69aebc2d25 [branch-2.1] Picks "[Fix](schema change) Fix can't do reorder column schema change for MOW table and duplicate key table #37067" (#37226)
## Proposed changes

picks https://github.com/apache/doris/pull/37067
2024-07-03 22:42:51 +08:00
84f5bb73da [refactor](nereids) refactor analyze view (#37106) (#37163)
The Analyzer of NereidsPlanner use different rules to analyze normal plan and view, to prevent the plans in the views analyze multiple times, because some rules can not apply multiple times, say, decimal type coercion, if this rule applied multiple times, it will generate a wrong result.

But this design is trick. Normally, after process the LogicalView, the whole plan tree in the LogicalView should not contains unbound plan, but the current situation is not like this. And this problem block development of some rules, so I refactor it:
1. the Analyzer will not traverse the children of the LogicalView
2. After link the LogicalView to the outer plan tree, the whole plan tree of the LogicalView will not contains unbound plan
3. analyze view and table use the same rules, keep it simple
2024-07-03 19:09:49 +08:00
45fc1c7182 [opt](hive) save hive table schema in transaction for 2.1 (#37127)
## Proposed changes

pick #37008
2024-07-03 17:32:58 +08:00
b3f2bd20e3 [feat](nereids) support explain delete from clause #36782 (#37100)
## Proposed changes
pick from  #36782

support explain like:
explain delete from T where A=1

Issue Number: close #xxx

<!--Describe your changes.-->

(cherry picked from commit dc369cd13096dbb90700f7fbf8f35a9059d9906f)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-03 15:08:08 +08:00
5969d6521f [branch-2.1](function) fix nereids fold constant wrong result of abs (#37065) (#37108)
pick https://github.com/apache/doris/pull/37065
2024-07-03 11:58:06 +08:00
fb642d0227 [Fix](hive-writer) Fixed the issue where uncompletedMpuPendingUploads did not remove objects correctly. (#37173)
Backport #36905.
2024-07-03 11:09:46 +08:00
4761090848 [fix](tvf) Partition columns in CTAS need to be compatible with the STRING type of external tables/TVF (#37161)
bp: #35489
2024-07-03 10:58:08 +08:00
17056dcdb9 [Fix](multi-catalog) Fix the transaction is not removed in abnormal situations by removing transaction in finally block. (#37172)
Backport #36705.
2024-07-03 10:56:41 +08:00
0aeb768bf9 [Fix](export/outfile) Support compression when exporting data to Parquet / ORC. (#37167)
bp: #36490
2024-07-03 10:53:57 +08:00
6804ae5827 [enhance](mtmv)reduce the behavior of triggering the mtmv state to change to schema_change (#36513) (#37122)
pick from https://github.com/apache/doris/pull/36513
2024-07-03 10:42:03 +08:00
03942f23f6 [fix](mtmv)fix when related table drop partition,mv partition is sync… (#36602)
pick: https://github.com/apache/doris/pull/36547
2024-07-02 23:08:58 +08:00
5bb6642b86 [fix](mtmv)mtmv support default key (#36221) (#36601)
pick : https://github.com/apache/doris/pull/36221
2024-07-02 23:08:38 +08:00
fbca3196c5 [enhance](mtmv) not allow modify data of MTMV (#35870) (#37129)
pick: https://github.com/apache/doris/pull/35870
2024-07-02 23:06:45 +08:00
0a1abf10d6 [fix](auth)support check priv when tvf use resource (#36928) (#37132)
pick https://github.com/apache/doris/pull/36928
2024-07-02 23:03:29 +08:00
177764647d [fix](mtmv)fix mtmv dead lock (#37009) (#37133)
pick https://github.com/apache/doris/pull/37009
2024-07-02 23:01:31 +08:00
6716f9e736 [fix](mtmv)fix MTMVTask log has null pointer (#36840) (#37131)
pick https://github.com/apache/doris/pull/36840
2024-07-02 23:01:15 +08:00
65375b48fc [enhance](mtmv)support replace materialized view (#36749) (#37147)
pick: https://github.com/apache/doris/pull/36749
2024-07-02 23:00:55 +08:00
b3eaf0e4d2 [bugfix](hive)Prevent multiple fs from being generated for 2.1 (#37142)
pick #36954
2024-07-02 22:54:40 +08:00
bd24a8bdd9 [Fix](csv_reader) Add a session variable to control whether empty rows in CSV files are read as NULL values (#37153)
bp: #36668
2024-07-02 22:12:17 +08:00
e25717458e [opt](catalog) add some profile for parquet reader and change meta cache config (#37040) (#37146)
bp #37040
2024-07-02 20:58:43 +08:00
42f4271e9d [enhancement](nereids) speedup sql cache with variable (#37090) (#37119)
cherry pick from #37090
2024-07-02 19:25:22 +08:00
b63e6ac350 [fix](ES Catalog)Add array types support in esquery function (#36936) (#37054)
backport #36936
2024-07-02 17:48:35 +08:00
7443e8fcf2 [cherry-pick](branch-2.1) fix single compaction test p2 #34568 #36881 (#37075) 2024-07-02 15:22:04 +08:00
496024ffb5 [fix](mtmv) mtmv add case and fix bug (#36019) (#36600)
pick from: #36019
2024-07-02 14:24:53 +08:00
ebc1afed4a [enhance](mtmv)when calculating the availability of MTMV, no longer c… (#36617)
pick: #36507
2024-07-02 14:24:33 +08:00
0c8934642e [opt](nereids)using mv's derived stats #35721 (#37098)
pick from #35721

(cherry picked from commit a481d79b1e258f7b6711157bb0d5e4c224e680e5)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-02 13:54:05 +08:00
a9764b9d39 [chore](multi catalog) Print serde properties when show create hive-external-table (#34966) (#37096)
bp #34966

Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com>
2024-07-02 11:07:10 +08:00
011f203d71 [opt](log) Remove unnecessary warning log (#37093)
When enable_profile = true or report_succeed=true, it is very likely
fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java::reportExecStatus
will print much warning log. It is not necesary.
2024-07-02 10:53:36 +08:00
3f382b797a [branch-2.1][improvement](sqlserver catalog) Configurable whether to use encrypt when connecting to SQL Server using the catalog (#36971)
pick (#36659)
pick #37015
In previous versions, we used druid as the default JDBC connection pool,
which can use custom decryption to parse the certificate when SQL Server
encryption is turned on. However, in the new version, after changing
HikariCP as the default connection pool, the SQLServer certificate
cannot be parsed, so encryption needs to be turned off for normal use.
Therefore, a parameter is added to decide whether to disable SQLServer
encryption. It is not disabled by default.
2024-07-02 10:14:43 +08:00
6425ce8a89 [Fix](Prepared Statment) use fixed charset to init StringLiteral (#37084)
picked from #36860
2024-07-01 23:11:13 +08:00
e686e85f27 [opt](split) add max wait time of getting splits (#36842)
bp: #36843
2024-07-01 22:05:25 +08:00
89b4918e88 [branch-2.1](stmt) fix show create table consistency #37074 (#37079)
pick #37074
2024-07-01 20:35:05 +08:00
24d236b210 [feat](Nereids) Optimize Sum Literal Rewriting by Excluding Single Instances (#35559) (#37047)
pick from master #35559

This PR introduces a change in the method removeOneSumLiteral to enhance
the performance of sum literal rewriting in SQL queries. The
modification ensures that sum literals appearing only once, such as in
expressions like select count(id1 + 1), count(id2 + 1) from t, are not
rewritten.
2024-07-01 14:57:15 +08:00
14c991f09b [Feature](Prepared Statement) fix and enable enable_server_side_prepared_statement by default #36581 (#36818)
picked from #36581
2024-07-01 14:35:17 +08:00
62c4451c97 [branch-2.1][improvement](jdbc catalog) Modify the maximum number of connections in the connection pool to 30 by default (#37023)
pick (#36720)

In many cases, we found that users would use JDBC Catalog to perform a
large number of queries, which resulted in the maximum of 10 connections
being insufficient, so I adjusted it to 30, which covered most needs.
2024-07-01 12:22:20 +08:00
12dddfc26c [branch-2.1] Pick "[Fix](autoinc) try fix concurrent load problem with auto inc column #36421" (#37027)
## Proposed changes

pick https://github.com/apache/doris/pull/36421
2024-06-30 13:10:03 +08:00
e25b0d7c37 [branch-2.1][improvement](mysql catalog) disable mysql AbandonedConnectionCleanupThread (#36970)
pick (#36655)
2024-06-29 18:35:41 +08:00
a9855ca173 [Improvement](set) enable admin_set_frontend_config can apply to all fe (#37022)
bp #34685
2024-06-29 09:17:38 +08:00
4bec08c7e8 [fix](inverted index)Make build index operation only affect base index (#36869) (#36988)
backport #36869
2024-06-28 16:57:07 +08:00
72de9c1b8f [fix](autoinc) avoid duplicated auto inc when role of fe changes (#36961)
## Proposed changes

pick #36960 

Issue Number: close #xxx

<!--Describe your changes.-->
2024-06-28 16:54:48 +08:00
b46c86f352 [improvement](partition rebalance) improve partition rebalance choose candidate speed #36509 (#36976)
cherry pick from #36509
2024-06-28 16:31:50 +08:00