Commit Graph

8838 Commits

Author SHA1 Message Date
3d6689d32c [fix](restore) Reset next version for restored partitions (#38343)
Cherry-pick #38321
2024-07-25 16:07:54 +08:00
32530113cf [cherry-pick](branch-2.1) Fix some group commit forward to master problems (#38345)
## Proposed changes

<!--Describe your changes.-->

Pick #38228 #38265
2024-07-25 14:48:02 +08:00
a8139d9f39 [Improvement](profile) Provide more info for schedule time (#38290) (#38338)
## Proposed changes

pick #38290

<!--Describe your changes.-->
2024-07-25 14:43:52 +08:00
e396f853a0 Pick "[enhance](Cooldown) Use config to control whether use cooldown replica for scanning first" (#38322)
## Proposed changes

<!--Describe your changes.-->

Same as master #37492
2024-07-25 12:17:38 +08:00
c23228e331 [Improvement](profile) add catalog info in profile (#38302)
bp #38283
2024-07-25 11:57:42 +08:00
bc7fc4106d [branch-2.1](function) fix FE impl of some time functions (#37746) (#38316)
pick https://github.com/apache/doris/pull/37746

before:
```sql
mysql> select date_ceil("2020-12-12 12:12:12.123", interval 2 second);
+-----------------------+
| '2020-12-12 12:12:12' |
+-----------------------+
| 2020-12-12 12:12:12   |
+-----------------------+
1 row in set (0.10 sec)

mysql> select CONVERT_TZ('9999-12-31 23:59:59.999999', 'Pacific/Galapagos', 'Pacific/Galapagos');
+------+
| NULL |
+------+
| NULL |
+------+
1 row in set (0.09 sec)

mysql [(none)]>select CONVERT_TZ('9999-12-31 23:59:59.999999', 'Pacific/Galapagos', 'Pacific/GalapaGoS');
+-----------------------------------------------------------------------------------------------------------+
| convert_tz(cast('9999-12-31 23:59:59.999999' as DATETIMEV2(6)), 'Pacific/Galapagos', 'Pacific/GalapaGoS') |
+-----------------------------------------------------------------------------------------------------------+
| 9999-12-31 23:59:59.999999                                                                                |
+-----------------------------------------------------------------------------------------------------------+
1 row in set (0.08 sec) --- gone to BE
```
after:
```sql
mysql> select date_ceil("2020-12-12 12:12:12.123", interval 2 second);
+------------------------------+
| '2020-12-12 12:12:14.000000' |
+------------------------------+
| 2020-12-12 12:12:14          |
+------------------------------+
1 row in set (0.11 sec)

mysql> select CONVERT_TZ('9999-12-31 23:59:59.999999', 'Pacific/Galapagos', 'Pacific/Galapagos');
+-----------------------------------------------------------------------------------------------------------+
| convert_tz(cast('9999-12-31 23:59:59.999999' as DATETIMEV2(6)), 'Pacific/Galapagos', 'Pacific/Galapagos') |
+-----------------------------------------------------------------------------------------------------------+
| 9999-12-31 23:59:59.999999                                                                                |
+-----------------------------------------------------------------------------------------------------------+
1 row in set (0.23 sec)

mysql> select CONVERT_TZ('9999-12-31 23:59:59.999999', 'Pacific/Galapagos', 'Pacific/GalapaGoS');
+------------------------------+
| '9999-12-31 23:59:59.999999' |
+------------------------------+
| 9999-12-31 23:59:59.999999   |
+------------------------------+
1 row in set (0.11 sec) --- finished in FE
```
2024-07-25 11:38:27 +08:00
8ad4390edb [fix](nereids) refine row count estimation for mark join (#38270) (#38297)
pick from master #38270
2024-07-25 10:19:13 +08:00
e23c1339a8 [fix](group commit) Fix prepare stmt setNull return too many filtered rows error (#38262) (#38276)
Pick https://github.com/apache/doris/pull/38262
2024-07-24 19:02:59 +08:00
17c33665e5 [fix](agg_state) adjust nullable should apply on agg_state inner type too (#37489) (#38281)
pick from master #37489

after adjust nullable, some children nullable has changed. so, we need
to update agg_state type inner type nullable too.
2024-07-24 13:55:09 +08:00
792bd7c74a [cherry-pick](branch-2.1) Pick "[Enhancement](group commit)Optimize be select for group commit #35558" (#37830)
Pick #35558
2024-07-24 09:21:07 +08:00
798b8573c9 [fix](load data) decommission replica don't load data when it misses versions #38198 (#38256)
cherry pick from #38198
2024-07-23 20:28:05 +08:00
b591837e13 [fix](insert) Pick Insert init plan error may forget to abort txn (#34220) (#38260)
Pick https://github.com/apache/doris/pull/34220
2024-07-23 20:21:38 +08:00
ca7d8325af [Fix](Nereids) TopN should forbid two-phase topN if child is DistributionSpecGather (#36877) (#38164)
pick from master #36877
2024-07-23 16:27:55 +08:00
ba5d23aea8 [enhancement](log) Changed log of waiting previous transactions to info level for debug convenience (#38133) (#38230)
## Proposed changes

As title.
2024-07-23 14:16:04 +08:00
33e04b12ff [conf](parallel) Reduce parallel tasks for large cluster (#38196) (#38233)
For large cluster, too many parallel tasks will cause performance issue.
So this PR limit the max parallel tasks in Doris.

pick #38196

<!--Describe your changes.-->
2024-07-23 13:42:31 +08:00
fdc8b454d2 [chore](backup) Remove delete_if_exists properties for creating repository (#38192)
delete_if_exists is a temporary solution introduced in #25847, to avoid
concurrent testing conflicts.

Cherry-pick #38190
2024-07-23 11:09:02 +08:00
41b355721f [fix](multicatalog) make lastdbofcatalog a session variable (#38114)
bp #37828
2024-07-23 10:34:40 +08:00
193be20c86 [feature](csv)Supports reading CSV data using LF and CRLF as line separators. (#37687) (#38099)
bp #37687
2024-07-22 22:53:04 +08:00
87306858a1 [Fix](JobManager)Release the lock immediately after modifying job metadata to avoid holding the lock for an extended period.#38162 (#38163)
## Proposed changes
#38162

<!--Describe your changes.-->
2024-07-20 10:52:21 +08:00
be3bccf04b [Fix](ranger)Resolving JSR 311 conflicts (#37575) (#38109)
## Proposed changes
### log
```
	at org.apache.doris.DorisFE.main(DorisFE.java:95) ~[doris-fe.jar:1.2-SNAPSHOT]
2024-07-09 11:56:53,826 WARN (mysql-nio-pool-0|316) [ConnectProcessor.handleQueryException():524] Process one query failed because unknown reason:
java.lang.NoSuchMethodError: 'void javax.ws.rs.core.NewCookie.<init>(java.lang.String, java.lang.String, java.lang.String, java.lang.String, int, java.lang.String, int, java.util.Date, boolean, boolean)'
	at org.glassfish.jersey.message.internal.CookiesParser$MutableNewCookie.getImmutableNewCookie(CookiesParser.java:130) ~[jersey-common-2.35.jar:?]
	at org.glassfish.jersey.message.internal.CookiesParser.parseNewCookie(CookiesParser.java:176) ~[jersey-common-2.35.jar:?]
	at org.glassfish.jersey.message.internal.HttpHeaderReader.readNewCookie(HttpHeaderReader.java:335) ~[jersey-common-2.35.jar:?]
	at org.glassfish.jersey.message.internal.NewCookieProvider.fromString(NewCookieProvider.java:87) ~[jersey-common-2.35.jar:?]
	at org.glassfish.jersey.message.internal.NewCookieProvider.fromString(NewCookieProvider.java:34) ~[jersey-common-2.35.jar:?]
	at javax.ws.rs.core.NewCookie.valueOf(NewCookie.java:126) ~[jsr311-api-1.1.1.jar:?]
	at com.sun.jersey.api.client.ClientResponse.getCookies(ClientResponse.java:783) ~[jersey-bundle-1.19.3.jar:1.19.3]
	at org.apache.ranger.admin.client.RangerAdminRESTClient.setCookieReceivedFromRoleDownloadSession(RangerAdminRESTClient.java:1364) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1220) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:167) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:183) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:123) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:495) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:144) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:245) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.doris.catalog.authorizer.ranger.hive.RangerHivePlugin.<init>(RangerHivePlugin.java:25) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.catalog.authorizer.ranger.hive.RangerHiveAccessController.<init>(RangerHiveAccessController.java:59) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.catalog.authorizer.ranger.hive.RangerHiveAccessControllerFactory.createAccessController(RangerHiveAccessControllerFactory.java:28) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.mysql.privilege.AccessControllerManager.createAccessController(AccessControllerManager.java:102) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.datasource.ExternalCatalog.initAccessController(ExternalCatalog.java:326) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.datasource.CatalogFactory.createCatalog(CatalogFactory.java:163) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.datasource.CatalogFactory.createFromStmt(CatalogFactory.java:91) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.datasource.CatalogMgr.createCatalog(CatalogMgr.java:250) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:381) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:2992) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:1019) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:619) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:537) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:385) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:241) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:260) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:288) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:342) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
```
### changes
JAX-RS (Java API for RESTful Web Services) is the implementation of the
JSR 311 specification, providing the Java API for creating RESTful web
services. JAX-RS 1.0 was the first version based on the JSR 311
specification. Subsequent versions of JAX-RS, such as JAX-RS 2.0, have
extended and improved upon the initial JSR 311 specification while
maintaining compatibility. JSR 311 is the specification, and JAX-RS is
the actual implementation of that specification. They share the same
package name, but when both coexist in a project, it can lead to loading
exceptions.

Therefore, I excluded the jsr311-api jar and retained only jax-rs.

(cherry picked from commit 400c1fd5e02ede895a036df34a7fdf54323e9865)

## Proposed changes

Issue Number: close #37575

<!--Describe your changes.-->
2024-07-20 00:21:59 +08:00
b24774c340 [branch-2.1][fix](compatibility) Enhance SSL and Long Flag Handling in MySQL Handshake (#38140)
pick(#38086)
Issue Number: close #38065

This PR introduces a fix to the SSL and CLIENT_LONG_FLAG handling in the
MySQL handshake process. It ensures that the flags are correctly set
according to the protocol requirements and resolves the issues related
to decimal data fetching when SSL is enabled.
2024-07-19 22:50:51 +08:00
dfc82a4011 [fix](inverted index)Support rename column with inverted index (#38079) (#38121)
## Proposed changes

backport #38079
2024-07-19 17:58:49 +08:00
27d7461644 [Opt](Iceberg) Simplify the code of getting time travel snapshotId (#34299) (#38101)
bp #34299

Co-authored-by: Butao Zhang <zhangbutao@cmss.chinamobile.com>
2024-07-19 09:45:56 +08:00
bb2b7774df [feature](iceberg) iceberg write support insert overwrite and optimize hive write transaction statistics and (#37191) (#38097)
bp #37191

Co-authored-by: kang <35803862+ghkang98@users.noreply.github.com>
Co-authored-by: lik40 <lik40@chinatelecom.cn>
2024-07-19 09:45:41 +08:00
301ff6af22 [Fix](nereids) add backquote and qualifier to udf, column, column alias and table alias when create view (#37237) (#37984)
cherry-pick #37237 to branch-2.1

---------

Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2024-07-19 00:56:26 +08:00
5d17ff9d95 [fix](mtmv)Mtmv support force replica 2.1 (#38090)
master do not have bug, because has call
`PropertyAnalyzer.getInstance().rewriteOlapProperties`

2.1 not have this method ,should call
`PropertyAnalyzer.rewriteReplicaAllocationProperties`
2024-07-18 23:52:34 +08:00
426493a7f1 [cherrypick] Fix profile eject (#38095)
pick https://github.com/apache/doris/pull/38030
2024-07-18 23:38:56 +08:00
6733c7c226 [chore](fe) Returns dropped partitions in GetMeta request (#38069)
Cherry-pick #37196, #37326.

The CCR syncer needs to know the distribution of tables, partitions,
indexes, and replicas when synchronizing binlogs. If a partition is
deleted before the binlog synchronization is complete, the CCR syncer
cannot continue synchronizing. This PR will record the deleted
partitions and include them in the get meta response, allowing the CCR
syncer to filter out the binlogs that belong to these partitions.

The CCR syncer part PR is selectdb/ccr-syncer#117.
2024-07-18 21:20:57 +08:00
4f03bde1e8 [Feat](job)scheduled job allows the time to be set in the past (#36732) (#37573)
## Proposed changes

In some cases, users want to set the start time to a time in the past.

**For periodic scheduling tasks only, the start time of a one-time task
must be set to the future or present time**
**If the start time is the past time, then the next execution time
should be the start time + time interval.**

(cherry picked from commit fa6061070f6d4063b1966e99fc285e4e6b9c3750)



#36732
2024-07-18 19:15:10 +08:00
c30c1d2436 [branch-2.1] Picks "[opt](delete) Delete job should retry for failure that is not DELETE_INVALID_XXX #37834" (#38032)
## Proposed changes

picks https://github.com/apache/doris/pull/37834 and
https://github.com/apache/doris/pull/38043
2024-07-18 14:50:30 +08:00
7dd01b1fa9 [fix](publish) Pick Catch exception in genPublishTask to make one failed txn does not block the other txns (#37724) (#38042)
Pick https://github.com/apache/doris/pull/37724
2024-07-18 14:15:49 +08:00
ebc178af23 [fix](nereids)acos function should return null literal instead of NaN value (#37996)
pick from master https://github.com/apache/doris/pull/37932

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-18 09:28:56 +08:00
dcfc72bb36 [fix](nereids) bug: after is-null stats derive, other column stats are dropped (#37809) (#38024)
## Proposed changes
pick from #37809 
Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-18 08:11:09 +08:00
b16cd30a20 [opt](inverted index) set support_phrase default true if parser is set (#38028)
## Proposed changes

https://github.com/apache/doris/pull/37949

<!--Describe your changes.-->
2024-07-18 08:07:16 +08:00
629d7ff048 [fix](nereids) fix aggr node colocate flag local shuffle depends on (#38016)
## Proposed changes

pick from https://github.com/apache/doris/pull/37788

Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
2024-07-17 21:34:30 +08:00
78d51ca320 [fix](Nereids) tablet prune wrong when decimal value scale is nagtive (#37889) (#38013)
pick from master #37889

we use unscaled value of BigDecimal in tablet prune. So we need to
ensure BigDecimal's precision and scale is same with the literal
contains it.
2024-07-17 20:12:14 +08:00
7180290d68 [fix](dynamic partition) fix create dynamic partition with overlap not throw exception #37924 (#37964)
cherry-pick:  #37924
2024-07-17 19:13:35 +08:00
2e6f49af15 [opt](Nereids) support no-key hint parameter (#37720) (#37988)
pick from master #37720

support hint use parameter without key, like:

```sql
SELECT /*+ query_timeout(3000) */ * FROM t;
```
2024-07-17 16:59:25 +08:00
b2a4cff51b [fix](nereids)fix nullable property of ForEachCombinator (#37980)
## Proposed changes

pick from master https://github.com/apache/doris/pull/37796

<!--Describe your changes.-->
2024-07-17 13:48:21 +08:00
f2cf8b7d6f [Fix](InternalSchema) Compute nodes should not be used for Internal schema three replica (#36130) (#37961)
bp #36130

Co-authored-by: HB <137497191@qq.com>
Co-authored-by: camby <104178625@qq.com>
2024-07-17 13:45:32 +08:00
ba66ff5768 [fix](multi-catalog)fix paimon meta properties convert (#37249) (#37958)
bp #37249

Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
2024-07-17 01:04:33 +08:00
dbd2b0abf7 [improvement](mtmv) improve mv rewrite performance by reuse the shuttled expression (#37197) (#37935)
## Proposed changes

chrry-pick 2.1
pr: https://github.com/apache/doris/pull/37197
commitId: 701c7db4
2024-07-17 00:52:58 +08:00
f7068b5658 [cherry-pick](branch-2.1) Make doris read hive text table parameters and behavior consistent with hive (#37840)
## Proposed changes

pick from master https://github.com/apache/doris/pull/37638

<!--Describe your changes.-->
2024-07-16 22:24:50 +08:00
49faccbf50 [enhancement](nereids) speedup sql cache with use variable as partition predicate (#37943)
follow up #37090

support reuse sql cache when use variable as partition predicate and variable change:
```sql
set @dt='2024-07-16';
-- create cache 1
select * from tbl where dt = @dt;

set @dt='2024-07-17';
-- create cache 2, will not invalidate cache 1
select * from tbl where dt = @dt;

set @dt='2024-07-16';
-- reuse cache 1
select * from tbl where dt = @dt;
```
2024-07-16 22:11:18 +08:00
2edb9501b5 [Pick]Fix show role stmt missing grouo info (#37920)
## Proposed changes

pick #36032
2024-07-16 20:41:59 +08:00
adf11737f0 [fix](catalog)Fix internal program error causing client to get stuck … (#37821)
…(#37551)

pick: https://github.com/apache/doris/pull/37551
2024-07-16 20:22:47 +08:00
9ff129b630 [fix](stream_load) fix stream load may failed caused by column name with keyword (#35822) (#37890)
#35938 #35822 
let 
KW_SQL,
KW_CACHE,
KW_COLOCATE,
KW_COMPRESS_TYPE,
KW_DORIS_INTERNAL_TABLE_ID,
KW_HOTSPOT,
KW_PRIVILEGES,
KW_RECENT,
KW_STAGES,
KW_WARM,
KW_UP,
KW_CONVERT_LSC
be as non-reserved

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

---------

Co-authored-by: caiconghui1 <caiconghui1@jd.com>
2024-07-16 20:20:34 +08:00
faa425bee5 [fix](colocate) fix colocate join while multi tables (#37729) (#37859)
cherry-pick #37729 to branch-2.1
2024-07-16 19:02:36 +08:00
81a7542cae [pick]Add audit log event queue size limit (#37914)
## Proposed changes
pick #37786
2024-07-16 19:00:22 +08:00
638d4b6f27 [Bug](function) Fix function for cast string as date/datetime (#35637) (#37891)
## Proposed changes

Issue Number: close #35635
cherry-pick https://github.com/apache/doris/pull/35637 from master to
branch-2.1

<!--Describe your changes.-->
Cast rules:Consistent with mysql.
String:Date
The first part is 1-digit x: 000x-month-day
The first part is 2-digit xy: 20xy-month-day / 19xy-month-day The first
part is 3-digit xyz: 20xy-0z-day / 19xy-0z-day The first part is 4-digit
xyzw: xyzw-month-day

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 17:22:03 +08:00