Commit Graph

19515 Commits

Author SHA1 Message Date
d19b3a5cb6 [fix](function) the bucket number arg of width_bucket should be a positive integer value (#37892) (#38295)
## Proposed changes

pick #37892
2024-07-25 11:56:47 +08:00
bc7fc4106d [branch-2.1](function) fix FE impl of some time functions (#37746) (#38316)
pick https://github.com/apache/doris/pull/37746

before:
```sql
mysql> select date_ceil("2020-12-12 12:12:12.123", interval 2 second);
+-----------------------+
| '2020-12-12 12:12:12' |
+-----------------------+
| 2020-12-12 12:12:12   |
+-----------------------+
1 row in set (0.10 sec)

mysql> select CONVERT_TZ('9999-12-31 23:59:59.999999', 'Pacific/Galapagos', 'Pacific/Galapagos');
+------+
| NULL |
+------+
| NULL |
+------+
1 row in set (0.09 sec)

mysql [(none)]>select CONVERT_TZ('9999-12-31 23:59:59.999999', 'Pacific/Galapagos', 'Pacific/GalapaGoS');
+-----------------------------------------------------------------------------------------------------------+
| convert_tz(cast('9999-12-31 23:59:59.999999' as DATETIMEV2(6)), 'Pacific/Galapagos', 'Pacific/GalapaGoS') |
+-----------------------------------------------------------------------------------------------------------+
| 9999-12-31 23:59:59.999999                                                                                |
+-----------------------------------------------------------------------------------------------------------+
1 row in set (0.08 sec) --- gone to BE
```
after:
```sql
mysql> select date_ceil("2020-12-12 12:12:12.123", interval 2 second);
+------------------------------+
| '2020-12-12 12:12:14.000000' |
+------------------------------+
| 2020-12-12 12:12:14          |
+------------------------------+
1 row in set (0.11 sec)

mysql> select CONVERT_TZ('9999-12-31 23:59:59.999999', 'Pacific/Galapagos', 'Pacific/Galapagos');
+-----------------------------------------------------------------------------------------------------------+
| convert_tz(cast('9999-12-31 23:59:59.999999' as DATETIMEV2(6)), 'Pacific/Galapagos', 'Pacific/Galapagos') |
+-----------------------------------------------------------------------------------------------------------+
| 9999-12-31 23:59:59.999999                                                                                |
+-----------------------------------------------------------------------------------------------------------+
1 row in set (0.23 sec)

mysql> select CONVERT_TZ('9999-12-31 23:59:59.999999', 'Pacific/Galapagos', 'Pacific/GalapaGoS');
+------------------------------+
| '9999-12-31 23:59:59.999999' |
+------------------------------+
| 9999-12-31 23:59:59.999999   |
+------------------------------+
1 row in set (0.11 sec) --- finished in FE
```
2024-07-25 11:38:27 +08:00
21b3fc3d1e [branch-2.1](function) fix coredump for MULTI_MATCH_ANY (#37959) (#38314)
pick https://github.com/apache/doris/pull/37959

[INVALID_ARGUMENT][E33] Compile regexp expression failed. got Embedded
start anchors not supported.. some expressions may be illegal
2024-07-25 11:34:22 +08:00
79a6496bb6 [branch-2.1](function) fix wrong result when convert_tz is out of bound (#37358) (#38313)
## Proposed changes

pick https://github.com/apache/doris/pull/37358

before:
```sql
mysql> select CONVERT_TZ(cast('0000-01-01 00:00:00.00001'  as DATETIMEV1), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533)));
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| convert_tz(cast('0000-01-01 00:00:00.00001' as DATETIME), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533))) |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| q535-12-31 08:01:19                                                                                                                               |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.12 sec)
```
now:
```sql
mysql> select CONVERT_TZ(cast('0000-01-01 00:00:00.00001'  as DATETIMEV1), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533)));
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| convert_tz(cast('0000-01-01 00:00:00.00001' as DATETIME), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533))) |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| NULL                                                                                                                                              |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.09 sec)
```
2024-07-25 11:32:44 +08:00
3ea26a8c95 [fix](external) record not found file number (#38253) (#38285)
bp #38253
2024-07-25 11:03:19 +08:00
8ad4390edb [fix](nereids) refine row count estimation for mark join (#38270) (#38297)
pick from master #38270
2024-07-25 10:19:13 +08:00
9a40cd5f9f [fix](compaction) fix duplicate key in agg/mor table caused by ordered data compaction (#38224) (#38299)
pick master #38224
2024-07-24 19:03:40 +08:00
e23c1339a8 [fix](group commit) Fix prepare stmt setNull return too many filtered rows error (#38262) (#38276)
Pick https://github.com/apache/doris/pull/38262
2024-07-24 19:02:59 +08:00
ef00dad680 [Fix](multi-catalog) Fix some undefined behaviors. (#38274)
## Proposed changes

backport #37845
2024-07-24 16:14:34 +08:00
c6cd1e2e3d [opt](inverted index) opt value extraction from column to string (#38179)
## Proposed changes

https://github.com/apache/doris/pull/37395
2024-07-24 15:38:41 +08:00
10c5c336d8 [branch-2.1](arrow-flight-sql) Add config arrow_flight_result_sink_buffer_size_rows (#38223)
pick #38221
2024-07-24 15:15:39 +08:00
bb2bc77717 [Pick 2.1](inverted index) skip index compaction for inverted index V2 (#38278)
Enabling index compaction in the inverted index V2 format currently
causes unexpected errors, especially in the case of tables with hybrid
indexes, such as BKD index and Fulltext index together.

backport #38209
2024-07-24 14:38:25 +08:00
17c33665e5 [fix](agg_state) adjust nullable should apply on agg_state inner type too (#37489) (#38281)
pick from master #37489

after adjust nullable, some children nullable has changed. so, we need
to update agg_state type inner type nullable too.
2024-07-24 13:55:09 +08:00
792bd7c74a [cherry-pick](branch-2.1) Pick "[Enhancement](group commit)Optimize be select for group commit #35558" (#37830)
Pick #35558
2024-07-24 09:21:07 +08:00
Pxl
cf2120a44a [Chore](brpc) add gc for abafreelist to avoid eagain and set brpc tim… (#38177)
pick from #37888
2024-07-23 21:24:36 +08:00
c73d40f869 [Fix](test)Fix the job's erroneous test case for asynchronous tests of periodic tasks. (#38264)
…

Since there might be delays in execution, taskCount should be set to
>=1.

## Proposed changes

Issue Number: close #xxx

#38263
2024-07-23 21:22:16 +08:00
62d2f0a05e [fix](mtmv)Fix mtmv name to resolve conflicts in regression test (#38056)
pick from master #36902 

## Proposed changes

[fix](mtmv)Fix mtmv name to resolve conflicts

Co-authored-by: Dongyang Li <hello_stephen@qq.com>
2024-07-23 20:57:29 +08:00
798b8573c9 [fix](load data) decommission replica don't load data when it misses versions #38198 (#38256)
cherry pick from #38198
2024-07-23 20:28:05 +08:00
b591837e13 [fix](insert) Pick Insert init plan error may forget to abort txn (#34220) (#38260)
Pick https://github.com/apache/doris/pull/34220
2024-07-23 20:21:38 +08:00
a04583c9a1 [enhancement](regression-test) schema reordering case to 2.1 (#38131)
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
2024-07-23 19:42:37 +08:00
ca7d8325af [Fix](Nereids) TopN should forbid two-phase topN if child is DistributionSpecGather (#36877) (#38164)
pick from master #36877
2024-07-23 16:27:55 +08:00
ba5d23aea8 [enhancement](log) Changed log of waiting previous transactions to info level for debug convenience (#38133) (#38230)
## Proposed changes

As title.
2024-07-23 14:16:04 +08:00
e28dcda0e2 [fix](regression test) fix test_report_version_missing due to set force_olap_table_replication_num #38211 (#38232)
cherry pick from #38211
2024-07-23 14:15:34 +08:00
33e04b12ff [conf](parallel) Reduce parallel tasks for large cluster (#38196) (#38233)
For large cluster, too many parallel tasks will cause performance issue.
So this PR limit the max parallel tasks in Doris.

pick #38196

<!--Describe your changes.-->
2024-07-23 13:42:31 +08:00
fdc8b454d2 [chore](backup) Remove delete_if_exists properties for creating repository (#38192)
delete_if_exists is a temporary solution introduced in #25847, to avoid
concurrent testing conflicts.

Cherry-pick #38190
2024-07-23 11:09:02 +08:00
41b355721f [fix](multicatalog) make lastdbofcatalog a session variable (#38114)
bp #37828
2024-07-23 10:34:40 +08:00
193be20c86 [feature](csv)Supports reading CSV data using LF and CRLF as line separators. (#37687) (#38099)
bp #37687
2024-07-22 22:53:04 +08:00
aff3f292a7 [fix](pipeline) Fix blocked tasks if query is canceled before opening (#38200)
## Proposed changes

pick #38206

<!--Describe your changes.-->
2024-07-22 18:22:22 +08:00
73199122c2 [enhancement](compaction) Control the parallelism for urgent compacton tasks (#37782) (#38189)
## Proposed changes

For some urgent compaction tasks, their submittion should take
parallelism into account.

Currently, we apply the control policy for data loading in specific.
Other source of urgent tasks are considered as eager.
2024-07-22 17:22:53 +08:00
ce1c268743 [Fix](load) Fix uninitiated load_type (not a valid value for type) of StreamLoadContext when operate_txn_2pc #38033 (#38187)
cherry pick from #38033
2024-07-22 17:20:03 +08:00
9eac4f2797 [regression](case) fix typo (#38142)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: stephen <hello-stephen@qq.com>
2024-07-22 14:26:30 +08:00
b6dd7bcc6d 2.1.5-rc02 (#38167)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-07-21 17:59:02 +08:00
2e79612320 [env](compile) compile failed caused by <immintrin.h> (#37068) (#38130)
pick from master #37068
2024-07-20 14:10:31 +08:00
7b74e8d7ab add 2.1.5-rc01 (#38102)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-07-20 11:36:06 +08:00
87306858a1 [Fix](JobManager)Release the lock immediately after modifying job metadata to avoid holding the lock for an extended period.#38162 (#38163)
## Proposed changes
#38162

<!--Describe your changes.-->
2024-07-20 10:52:21 +08:00
5ab038cd89 [test](mtmv)Add group by aggregate negative case (#38055)
pick from master #36562 

## Proposed changes

[test](mtmv)Add group by aggregate negative case
2024-07-20 10:14:25 +08:00
be3bccf04b [Fix](ranger)Resolving JSR 311 conflicts (#37575) (#38109)
## Proposed changes
### log
```
	at org.apache.doris.DorisFE.main(DorisFE.java:95) ~[doris-fe.jar:1.2-SNAPSHOT]
2024-07-09 11:56:53,826 WARN (mysql-nio-pool-0|316) [ConnectProcessor.handleQueryException():524] Process one query failed because unknown reason:
java.lang.NoSuchMethodError: 'void javax.ws.rs.core.NewCookie.<init>(java.lang.String, java.lang.String, java.lang.String, java.lang.String, int, java.lang.String, int, java.util.Date, boolean, boolean)'
	at org.glassfish.jersey.message.internal.CookiesParser$MutableNewCookie.getImmutableNewCookie(CookiesParser.java:130) ~[jersey-common-2.35.jar:?]
	at org.glassfish.jersey.message.internal.CookiesParser.parseNewCookie(CookiesParser.java:176) ~[jersey-common-2.35.jar:?]
	at org.glassfish.jersey.message.internal.HttpHeaderReader.readNewCookie(HttpHeaderReader.java:335) ~[jersey-common-2.35.jar:?]
	at org.glassfish.jersey.message.internal.NewCookieProvider.fromString(NewCookieProvider.java:87) ~[jersey-common-2.35.jar:?]
	at org.glassfish.jersey.message.internal.NewCookieProvider.fromString(NewCookieProvider.java:34) ~[jersey-common-2.35.jar:?]
	at javax.ws.rs.core.NewCookie.valueOf(NewCookie.java:126) ~[jsr311-api-1.1.1.jar:?]
	at com.sun.jersey.api.client.ClientResponse.getCookies(ClientResponse.java:783) ~[jersey-bundle-1.19.3.jar:1.19.3]
	at org.apache.ranger.admin.client.RangerAdminRESTClient.setCookieReceivedFromRoleDownloadSession(RangerAdminRESTClient.java:1364) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1220) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:167) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:183) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:123) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:495) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:144) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:245) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
	at org.apache.doris.catalog.authorizer.ranger.hive.RangerHivePlugin.<init>(RangerHivePlugin.java:25) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.catalog.authorizer.ranger.hive.RangerHiveAccessController.<init>(RangerHiveAccessController.java:59) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.catalog.authorizer.ranger.hive.RangerHiveAccessControllerFactory.createAccessController(RangerHiveAccessControllerFactory.java:28) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.mysql.privilege.AccessControllerManager.createAccessController(AccessControllerManager.java:102) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.datasource.ExternalCatalog.initAccessController(ExternalCatalog.java:326) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.datasource.CatalogFactory.createCatalog(CatalogFactory.java:163) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.datasource.CatalogFactory.createFromStmt(CatalogFactory.java:91) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.datasource.CatalogMgr.createCatalog(CatalogMgr.java:250) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:381) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:2992) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:1019) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:619) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:537) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:385) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:241) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:260) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:288) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:342) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
```
### changes
JAX-RS (Java API for RESTful Web Services) is the implementation of the
JSR 311 specification, providing the Java API for creating RESTful web
services. JAX-RS 1.0 was the first version based on the JSR 311
specification. Subsequent versions of JAX-RS, such as JAX-RS 2.0, have
extended and improved upon the initial JSR 311 specification while
maintaining compatibility. JSR 311 is the specification, and JAX-RS is
the actual implementation of that specification. They share the same
package name, but when both coexist in a project, it can lead to loading
exceptions.

Therefore, I excluded the jsr311-api jar and retained only jax-rs.

(cherry picked from commit 400c1fd5e02ede895a036df34a7fdf54323e9865)

## Proposed changes

Issue Number: close #37575

<!--Describe your changes.-->
2024-07-20 00:21:59 +08:00
7819c75e55 [fix](shuffle) Fix local exchange dependency blocking (#38160)
## Proposed changes

pick #38151

<!--Describe your changes.-->
2024-07-20 00:19:47 +08:00
947d7594c8 [fix](func)fix array_with_const with larger than max_array_size (#38152)
## Proposed changes
backport: https://github.com/apache/doris/pull/37495
Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-20 00:19:00 +08:00
ffc0d6884d [Fix](load) Fix the channel leak when close wait has been cancelled #38031 (#38125)
cherry pick from #38031
2024-07-19 22:58:54 +08:00
d9fd419e47 [Fix](JsonReader) fix json with duplicate key entry may result out of bound exception (#38147)
#38146
2024-07-19 22:53:02 +08:00
b24774c340 [branch-2.1][fix](compatibility) Enhance SSL and Long Flag Handling in MySQL Handshake (#38140)
pick(#38086)
Issue Number: close #38065

This PR introduces a fix to the SSL and CLIENT_LONG_FLAG handling in the
MySQL handshake process. It ensures that the flags are correctly set
according to the protocol requirements and resolves the issues related
to decimal data fetching when SSL is enabled.
2024-07-19 22:50:51 +08:00
22b9cc6eb2 [fix](mysql)fix mysql row buf (#38145)
## Proposed changes
backport: https://github.com/apache/doris/pull/37936
Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-19 22:18:42 +08:00
7b141ffde7 [pick]add min scan thread num for workload group's scan thread (#38123)
## Proposed changes

pick #38096
2024-07-19 18:43:05 +08:00
dfc82a4011 [fix](inverted index)Support rename column with inverted index (#38079) (#38121)
## Proposed changes

backport #38079
2024-07-19 17:58:49 +08:00
d4d1688df2 [improve](test)Using Awaitility instead of Thread.sleep (#37812) (#38108)
## Proposed changes

Awaitility: Waits until a specific condition is met, which makes the
tests more reliable and less prone to intermittent failures. It ensures
that the test only proceeds when the expected condition is true.
Thread.sleep: Introduces a fixed delay, which can lead to flaky tests.
If the condition is met sooner than the sleep time, the test
unnecessarily waits. Conversely, if the condition is met after the sleep
time, the test will fail.

(cherry picked from commit 8d249a2562b15825a06a53a5a530a532a0ca2454)

## Proposed changes

Issue Number: close #37812

<!--Describe your changes.-->
2024-07-19 15:32:03 +08:00
53ca312751 [cherry-pick](branch-2.1) Pick "[Fix](delete) Fix delete on value case due to compaction (#37917)" (#38116)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Pick #37917
2024-07-19 13:51:39 +08:00
27d7461644 [Opt](Iceberg) Simplify the code of getting time travel snapshotId (#34299) (#38101)
bp #34299

Co-authored-by: Butao Zhang <zhangbutao@cmss.chinamobile.com>
2024-07-19 09:45:56 +08:00
bb2b7774df [feature](iceberg) iceberg write support insert overwrite and optimize hive write transaction statistics and (#37191) (#38097)
bp #37191

Co-authored-by: kang <35803862+ghkang98@users.noreply.github.com>
Co-authored-by: lik40 <lik40@chinatelecom.cn>
2024-07-19 09:45:41 +08:00
4d03e288c2 [fix](case)fix 2.1 s3 case (#38088)
extends : https://github.com/apache/doris/pull/37995
2024-07-19 04:00:56 +08:00