741807bb22
[performance](move-memtable) only call _select_streams when necessary ( #35576 ) ( #37406 )
...
cherry-pick #35576
2024-07-10 22:20:23 +08:00
f58032f1da
[fix](dynamic partition) drop partition exclude history_partition_num #37539 ( #37570 )
...
cherry pick from #37539
---------
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com >
2024-07-10 22:09:00 +08:00
a54ba2ea63
[fix](partition) Fix test partition regression case unstabitily #37439 ( #37583 )
...
cherry pick from #37439
2024-07-10 22:08:15 +08:00
3d4ec81eeb
[fix](regression test) fix unstable test_alter_colocate_group #37544 ( #37609 )
...
cherry-pick: #37544
2024-07-10 21:51:46 +08:00
0a95757a4d
[opt](test) Optimize execution time of test_s3_load case #37562 ( #37612 )
...
cherry pick from #37562
2024-07-10 19:09:46 +08:00
a7416f6ff9
[fix](dump) do not report success if dumping fails ( #37510 )
...
## Proposed changes
pick https://github.com/apache/doris/pull/37508
Issue Number: close #xxx
<!--Describe your changes.-->
2024-07-10 16:26:14 +08:00
afcc6170f6
[fix](txn_manager) Add ingested rowsets to unused rowsets when removing txn ( #37417 )
...
Generally speaking, as long as a rowset has a version, it can be
considered not to be in a pending state. However, if the rowset was
created through ingesting binlogs, it will have a version but should
still be considered in a pending state because the ingesting txn has not
yet been committed.
This PR updates the condition for determining the pending state. If a
rowset is COMMITTED, the txn should be allowed to roll back even if a
version exists.
Cherry-pick #36551
2024-07-10 14:25:44 +08:00
f65d1c4011
[chore](ci) rm repeat function name ( #37579 )
...
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-07-10 11:44:32 +08:00
0cdb371624
[branch-2.1](memory) Disable Arrow Jemalloc step 2 ( #37556 )
...
pick #37533
2024-07-10 11:34:18 +08:00
a4aaace481
[Fix](branch-2.1) Fix partial update regression test ( #37560 )
...
Version 2.1 does not support partial update for variant type data, so we
delete this case.
2024-07-10 10:44:24 +08:00
87cd366636
[chore](dependencies)Remove Unnecessary Dependencies ( #37469 ) ( #37555 )
...
bp #37469
2024-07-10 09:33:43 +08:00
770e7d21a4
[dependencies](fe)upgrade paimon to 0.8.1 ( #37205 ) ( #37554 )
...
bp #37205
2024-07-10 09:32:33 +08:00
27283b2915
[branch-2.1](memory) Disable Arrow Jemalloc ( #37529 )
...
pick ##37528
2024-07-09 23:48:27 +08:00
90757a9283
[test](auth)add upgrade and downgrade compatibility test case ( #37523 )
...
## Proposed changes
[test](auth)add upgrade and downgrade compatibility test case
2024-07-09 21:13:54 +08:00
08b3b73c3a
[fix](regression) Fix regression run in cloud use non-root user #34340 ( #37279 )
...
cherry pick from #34340
2024-07-09 20:48:30 +08:00
db4d061a68
[fix](Nereids) null type result with alias name should keep alias name ( #37457 ) ( #37524 )
...
pick from master #37457
2024-07-09 20:46:51 +08:00
8ef83259ff
[fix](planner) fix bug of select stmt toSql( #37274 ) ( #37344 )
...
cherry-pick from master #37274
2024-07-09 20:34:57 +08:00
9b075bc873
[fix](nereids) derive column stats for 'expr and A is not null' ( #37235 ) ( #37498 )
...
pick from #37235
the algorithm for computing stats for "expr1 and expr2" predicate is as
following:
1. compute output stats of expr1 based on input stats. the result stats
is denoted by leftStats
2. compute stats of expr2 based on leftStats after step1, leftStats
should be normalized to avoid abnormal cases, such as ndv > rowCount or
numNulls > rowCount
Issue Number: close #xxx
<!--Describe your changes.-->
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-07-09 17:46:57 +08:00
5247e0ff3a
[fix](clone) Increase robustness for clone #36642 ( #37413 )
...
cherry pick from #36642
2024-07-09 17:18:14 +08:00
7f6b846f58
[improve](fe) Add a counter metric for recording large editlog write ( #37328 ) ( #37474 )
2024-07-09 17:16:31 +08:00
7cda8db020
[fix](load) The NodeChannel should be canceled when failed to add block #37500 ( #37527 )
...
cherry pick from #37500
2024-07-09 17:01:04 +08:00
f7f0c20f00
[branch-2.1](cgroup memory) Correct cgroup mem info cache ( #37440 )
...
pick #36966
Co-authored-by: Hongkun Xu <xuhongkun666@163.com >
2024-07-09 16:19:37 +08:00
005304953e
[performance](load) do not copy input_block in memtable ( #36939 ) ( #37407 )
...
cherry-pick #36939
2024-07-09 15:59:44 +08:00
81360cf897
[opt](test) shorten the external p0 running time ( #37320 ) ( #37473 )
...
bp #37320
2024-07-09 15:35:15 +08:00
19eb9decba
[chore](ci) add isCloudMode ( #37518 )
...
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
Co-authored-by: stephen <hello-stephen@qq.com >
2024-07-09 14:09:16 +08:00
9b500faa0c
[fix](create table) create table fail not write drop table editlog #37488 ( #37506 )
...
cherry pick from #37488
2024-07-09 13:44:43 +08:00
2d5eb55310
[cherry-pick](branch-2.1) Pick "[Enhancement](partial update) Add some partial update case ( #36385 )" ( #37382 )
...
#36385
2024-07-09 10:52:04 +08:00
8d46f0e138
[fix](manager)fix manager insterface case.( #37082 ) ( #37317 )
...
bp #37082
2024-07-09 10:42:22 +08:00
4426d6d80f
[fix](fe) Add check editlog size mechanism for backupJob ( #35653 ) ( #37466 )
...
* When creating a backupJob with huge of tables in a database, it can
cause backupJob editlog size over 2GB and bdbje will throw exception
because of ByteBuffer overflow
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-07-09 10:33:28 +08:00
ca0e44f83f
[fix](case) fix struct format out files ( #37350 ) ( #37499 )
...
bp #37350
2024-07-09 10:11:50 +08:00
3337c1bbe3
[[enhancement](compaction) adjust compaction concurrency based on compaction score and workload ( #37491 )
...
adjust compaction concurrency based on compaction score and workload
#36672
fix null pointer when retrieving CPU load average #37171
2024-07-09 09:56:35 +08:00
1e3ab0ff8c
[fix](group commit) Pick make group commit cancel in time ( #36249 ) ( #37404 )
...
pick https://github.com/apache/doris/pull/36249/
2024-07-09 09:25:11 +08:00
1a25270918
[fix](group commit) Pick Fix the incorrect group commit count in log; fix the core in get_first_block ( #36408 ) ( #37405 )
...
Pick https://github.com/apache/doris/pull/36408/
2024-07-09 09:24:43 +08:00
a6ec78ec5f
[fix](truncate) fix tablet invert index leaky #37334 ( #37410 )
...
cherry pick from #37334
2024-07-08 20:57:30 +08:00
0a103aa11f
[improve](json)improve json support empty keys #36762 ( #37351 )
2024-07-08 19:04:51 +08:00
16890feb9a
[chore](suite) Add target_sql_return_maparray ( #37421 )
...
Cherry-pick #37303
2024-07-08 18:58:55 +08:00
4c5a7b26e2
[chore](restore) Log partition visible version ( #37414 )
...
Cherry-pick #36920 .
2024-07-08 18:58:16 +08:00
5280e277e7
[chore](be) Acquire and check MD5 digest of the file to download ( #37418 )
...
Cherry-pick #35807 , #36621 , #36726
2024-07-08 18:55:35 +08:00
4f00b41796
[chore](suites) Allow change db name in get_ccr_body ( #37424 )
...
Cherry-pick #37324 .
2024-07-08 17:47:43 +08:00
494b54a5a5
[enhancement](trash) support skip trash, update trash default expire time ( #37170 ) ( #37409 )
...
cherry-pick #37170
2024-07-08 15:33:02 +08:00
68352c7c77
[fix](Nereids) constant folding for str_to_date on datev1 is wrong ( #37360 ) ( #37367 )
...
pick from master #37360
when enable datev1, we should return datev1/datetimev1
2024-07-08 15:21:12 +08:00
b058867e39
[cherry-pick](branch-2.1) Change the case to nonConcurrent ( #37357 )
...
## Proposed changes
pick from master [#37356 ](https://github.com/apache/doris/pull/37356 )
Issue Number: close #xxx
<!--Describe your changes.-->
2024-07-08 15:12:48 +08:00
fbc954e8be
[feat](mtmv) Support grouping_sets rewrite when query rewrite by materialized view ( #36056 ) ( #37436 )
...
cherry pick from master
pr: #36056
commitId: 569c9772
2024-07-08 15:06:16 +08:00
779a51570e
[opt](mtmv) Set query rewrite by materialized view default enable ( #35897 ) ( #36949 )
...
cherry pick from master
pr: #35897
commitId: 603fa82f
2024-07-08 14:29:38 +08:00
95dad14062
[chore](query) print query id when killed by timeout checker ( #37402 )
...
pick #36868
2024-07-08 11:26:29 +08:00
c66df8d9e6
[branch-2.1](load) fix no error url if no partition can be found ( #36831 ) ( #37401 )
...
## Proposed changes
pick #36831
before
```
Stream load result: {
"TxnId": 2014,
"Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
"Comment": "",
"TwoPhaseCommit": "false",
"Status": "Fail",
"Message": "[DATA_QUALITY_ERROR]Encountered unqualified data, stop processing",
"NumberTotalRows": 1,
"NumberLoadedRows": 1,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 1669,
"LoadTimeMs": 58,
"BeginTxnTimeMs": 0,
"StreamLoadPutTimeMs": 10,
"ReadDataTimeMs": 0,
"WriteDataTimeMs": 47,
"CommitAndPublishTimeMs": 0
}
```
after
```
Stream load result: {
"TxnId": 2014,
"Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
"Comment": "",
"TwoPhaseCommit": "false",
"Status": "Fail",
"Message": "[DATA_QUALITY_ERROR]too many filtered rows",
"NumberTotalRows": 1,
"NumberLoadedRows": 0,
"NumberFilteredRows": 1,
"NumberUnselectedRows": 0,
"LoadBytes": 1669,
"LoadTimeMs": 58,
"BeginTxnTimeMs": 0,
"StreamLoadPutTimeMs": 10,
"ReadDataTimeMs": 0,
"WriteDataTimeMs": 47,
"CommitAndPublishTimeMs": 0,
"ErrorURL": "http://XXXX:8040/api/_load_error_log?file=__shard_4/error_log_insert_stmt_c6461270125a615b-2873833fb48d56a3_c6461270125a615b_2873833fb48d56a3 "
}
```
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-07-08 10:41:33 +08:00
dd18652861
[branch-2.1](routine-load) make get Kafka meta timeout configurable ( #37399 )
...
pick #36619
2024-07-08 10:39:17 +08:00
af7b69da48
[fix](nereids) Pick the stop watch is not reset ( #37168 ) ( #37397 )
...
Pick https://github.com/apache/doris/pull/37168 and
https://github.com/apache/doris/pull/37095
2024-07-08 10:28:03 +08:00
70f46c12b3
[improve](group commit) Pick Modify group commit case and modify cancel status ( #35995 ) ( #37398 )
...
Pick https://github.com/apache/doris/pull/35995
2024-07-08 10:27:08 +08:00
969f7532d6
[fix](deps) fix NoSuchMethodError: newInstanceFromKeytab when use kerberos ( #37322 )
2024-07-08 10:19:02 +08:00
