Commit Graph

9414 Commits

Author SHA1 Message Date
8fedb03fdc branch-2.1: [fix](auth)Fix after fe restarting, external permissions are lost (#43307)
Cherry-picked from #43275

Co-authored-by: zhangdong <493738387@qq.com>
2024-11-06 14:54:59 +08:00
9d5ebd6f34 [fix](export) make sure the export task being executed after writing "create export job" edit log #43250 (#43291)
cherry-pick #43250
2024-11-06 09:47:14 +08:00
71e33d9208 [feat](nereids)disable join reorder if any table row count is not available #43000 (branch-2.1) (#43247)
### What problem does this PR solve?
pick #43000 

<!--
You need to clearly describe your PR in this part:

1. What problem was fixed (it's best to include specific error reporting
information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what
is it now, why was it modified, and what possible impacts might there
be.
3. What features were added. Why this function was added.
4. Which codes were refactored and why this part of the code was
refactored.
5. Which functions were optimized and what is the difference before and
after the optimization.

The description of the PR needs to enable reviewers to quickly and
clearly understand the logic of the code modification.
-->

<!--
If there are related issues, please fill in the issue number.
- If you want the issue to be closed after the PR is merged, please use
"close #12345". Otherwise, use "ref #12345"
-->
Issue Number: close #xxx

<!--
If this PR is followup a preivous PR, for example, fix the bug that
introduced by a related PR,
link the PR here
-->
Related PR: #xxx

Problem Summary:

### Check List (For Committer)

- Test <!-- At least one of them must be included. -->

    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No colde files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:

    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?

    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

- Release note

    <!-- bugfix, feat, behavior changed need a release note -->
    <!-- Add one line release note for this PR. -->
    None

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2024-11-05 18:29:50 +08:00
2e00fa475b [fix](persist) Fix the generated dropping tmp partition sql #43079 (#43170)
cherry pick from #43079
2024-11-05 17:44:31 +08:00
8eb6a2a7c1 [enhancement](Nereids) boost characterLiteralTypeCoercion (#42941) (#43092)
cherry pick from #42941
2024-11-04 11:50:44 +08:00
d1e63c5201 [improvement](external)add some improvements for external scan (#38946) (#43156)
bp #38946

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-11-04 09:40:08 +08:00
e3170d6319 [bugfix](hive)fix after insert overwrite hive table, data error for 2.1 (#43049) (#43127)
bp: #43049
2024-11-03 23:38:47 +08:00
9e7a1c9ae8 [fix](export) fix potential export concurrency issue #43109 (#43117)
bp #43109
2024-11-02 08:53:43 +08:00
c5374be1bc [cherry-pick](branch-2.1) Pick "[Feature](schema change) Support add column bitmap with default value bitmap_empty (#42331)" (#42702)
Pick #42331
2024-11-01 21:46:38 +08:00
03a614f945 [fix](Export) fix dead lock of parallel Exporting (#43084)
### What problem does this PR solve?
bp: #43083
2024-11-01 21:27:25 +08:00
6c3a273d46 [fix](Nereids) replay create function npe since connect context not exists (#43075)
nereids will parse function statement when replay create alias function,
when do parse, need to get sql mode and other variable. because replay
thread do not have connect context, NPE be thrown when parsing
2024-11-01 21:01:24 +08:00
47fa76e0f6 [2.1][fix](jdbc catalog) Optimize JDBC Connection Closing to Ensure Proper Resource Release (#43074)
bp #43059
2024-11-01 20:34:18 +08:00
cb25c40ddf [fix](export) fix concurrent modification issue with export job #43051 (#43052)
bp #43051
2024-11-01 17:19:29 +08:00
355170a921 [cherry-pick](branch2.1) impl scalar functions trim_in、ltrim_in and rtrim_in (#42641)
pick https://github.com/apache/doris/pull/41681
2024-11-01 09:55:50 +08:00
e85e6c89ba [fix](auditlog) add missing audit log fields and duplicate audit log error #42262 (#43015)
bp #42262
2024-11-01 08:24:55 +08:00
df2bd6d525 [improvement](statistics)Improve statisitcs insert into audit log. (#42983) (#43032)
backport: https://github.com/apache/doris/pull/42983
2024-11-01 07:47:40 +08:00
a5efe1c66e [improve](restore) Log create replica task progress #42350 (#42967)
cherry pick from #42350
2024-10-31 14:36:34 +08:00
c573351e4e [fix](tvf) fix FE cannot start when replay alter view from tvf (#42866)
bp: #40872
2024-10-31 14:15:48 +08:00
6c3d42e09a [cherry-pick](branch-21) cherry-pick pr about (#42488) (#42099) (#42055) (#42916)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-31 14:14:19 +08:00
cc30a7e78e [fix](test) fix some unstable external p0 test cases (#42685) (#42943)
cherry-pick #42685

Co-authored-by: daidai <2017501503@qq.com>
2024-10-31 12:36:03 +08:00
597bcb112b [fix](Nerieds) fix FoldConstantRuleOnFE too slow when CaseWhen clause nesting multiple layers (#42897) (#42907)
cherry pick from master #42897
2024-10-31 11:15:34 +08:00
cdd32d9582 [enhance](hive) support reading hive table with OpenCSVSerde #42257 (#42940)
cherry pick from #42257

Co-authored-by: Socrates <suxiaogang223@icloud.com>
2024-10-31 11:12:07 +08:00
d67007c72d [improvement](statistics)Add log for estimating hive table row count logic. (#42921) (#42931)
backport: https://github.com/apache/doris/pull/42921
2024-10-31 10:30:41 +08:00
0f9c2c2f0a [fix](export) remove export task executor in TransientTaskExecutor (#42880)
### What problem does this PR solve?

Problem Summary:

There is a memory leak on FE side when continue running `export`
command.
The `ExportTaskExecutor` instances will be added to
`TransientTaskManager` and never be removed.

This PR mainly changes:
1. Remove unused `ExportTaskRegister` in `Env`.
2. Remove `ExportTaskExecutor` from `TransientTaskManager` once export
job is finished or cancelled.

### Release note

Fix FE memory leak by removing `ExportTaskExecutor` from
`TransientTaskManager` once export job is finished or cancelled.
2024-10-31 09:57:40 +08:00
ae88d032db [chore](ddl) support force_enable_feature_binlog #41796 (#42926)
cherry pick from #41796
2024-10-31 09:53:45 +08:00
fce4695f37 [Configuration](transactional-hive) Add skip_checking_acid_version_file session var to skip checking acid version file in some hive envs. (#42111)(#42225) (#42939)
cherry-pick (#42111)(#42225)

---------

Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
2024-10-31 09:52:20 +08:00
0d008b5a43 [feat](nereids)disable join reorder if column stats is invalid #41790 (branch-2.1) (#42902)
## Proposed changes
pick #41790
Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-30 23:47:03 +08:00
2a6d6c15c5 [enhance](hive)hive event code optimization #42637 (#42876)
cherry pick from #42637

Co-authored-by: zhangdong <493738387@qq.com>
2024-10-30 12:52:21 +08:00
17d84dc88f [enhance](paimon)paimon scanner code optimization #42606 (#42875)
cherry pick from #42606

Co-authored-by: zhangdong <493738387@qq.com>
2024-10-30 12:51:59 +08:00
ba4d01d93c [branch-2.1](hint) add detail for replica missing msg (#42546) 2024-10-29 22:47:41 +08:00
233fad4815 [opt](variable) force update some variable by variable version (#41607) (#42643)
pick from master #41607

variable version:
000-100: doris-2.0.x
100-200: doris-2.1.x
200-300: doris-3.0.x

update variables

000:
nereids_timeout_second = 30 if original value is 5

100:
enable_nereids_dml = true
enable_nereids_dml_with_pipeline = true
enable_nereids_planner = true
enable_fallback_to_original_planner = true
enable_pipeline_x_engine = true
2024-10-29 15:57:14 +08:00
1538b82221 [Impl](Nereids) add nereids gc cost time and be fold const cost time to profile (#42007) (#42516)
pick:#42007
add Nereids GarbageCollect Time and Nereids BeFoldConst Time to profile

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-29 15:16:40 +08:00
d5c43c51b5 [branch-2.1](fe) Avoid interrupt daemon thread and use proper polling interval (#42210) (#42646)
pick https://github.com/apache/doris/pull/42210
2024-10-29 10:10:26 +08:00
73621c480a [fix](log) fix group commit warn log (#42403) (#42575)
pick https://github.com/apache/doris/pull/42403
2024-10-28 21:04:29 +08:00
0baca54200 [fix](Nereids) offset do more than once when have shuffle after limit (#42576) (#42577)
pick from master #42576

intro by #39316. it want to fix a problem intro by #36699. but forgot to
remove all wrong code in #36699.

after #39316, we should not set offset on exchange, when the exchange is
on the top of a limit with offset.
2024-10-28 19:48:13 +08:00
85674814eb [fix](query-forward) Fix forward query exception or stuck or potential query result loss (#41303) (#42369)
## Proposed changes

1. Fix forward query exception if no status code is set in master
execution. EOF may result in this status.

2. Fix forward query stuck due to no result packet sent to mysql
channel. Should use result packets from master.

3. Fix potential forward query result loss if follower can read status
change during query process. Should judge by the status once before
execution.

4. Add assertion for regression test.
2024-10-28 17:39:57 +08:00
0e63133c80 [Chore](job) Provides configuration of job execution queue size (#42253) (#42530)
When dealing with a large number of tasks, the default execution queue
size is 1024. This can lead to tasks being dropped if the queue becomes
full.
eg

`dispatch instant task failed, job id is xxx`

To address this, you can add the parameters `insert_task_queue_size` and
`mtmv_task_queue_size` in the `fe.conf` configuration file. These
parameters must be set to a power of 2.

**Keep in mind, increasing this value is recommended only when thread
resources are limited; otherwise, you should consider increasing the
number of task execution threads.**

(cherry picked from commit f9ea8f8229e9f5514c1773bd25c3cc11985c63fb)

## Proposed changes

Issue Number: #42253

<!--Describe your changes.-->
2024-10-28 13:42:08 +08:00
9eef393e2a [pick]support cgroup v2 (#42465)
## Proposed changes

pick #39991   #39374  #36663
2024-10-25 20:13:27 +08:00
4a62d9e44b Revert "[2.1][improvement](jdbc catalog) Add catalog property to enable jdbc connection pool" (#42481)
Reverts apache/doris#42255

We have found that after closing the connection pool, there will be
class loading problems and connection release problems for some data
sources. We will remove this function first and re-add it after solving
and testing it completely.
2024-10-25 19:37:36 +08:00
859332c918 [improve](common) Add synchronized to avoid concurrent modification #42384 (#42454)
cherry pick from #42384
2024-10-25 16:33:53 +08:00
b88d4db6e7 [fix](auth)Fix use encryptkey should check auth (#41791) (#42105)
pick from master #41791
2024-10-25 14:32:42 +08:00
120bf28d1e [fix](statistics)Skip shadow index while analyzing a table. (#42201) (#42414)
backport: https://github.com/apache/doris/pull/42201
2024-10-25 13:44:42 +08:00
db9c74c38f [improvement](statistics)Drop expired external stats only when the catalog is dropped. (#42244) (#42410)
backport: https://github.com/apache/doris/pull/42244
2024-10-24 23:47:53 +08:00
dba6296d3e [improvement](statistics)Skip auto analyze when table row count is not fully reported. (return -1) (#42209) (#42407)
backport: https://github.com/apache/doris/pull/42209
2024-10-24 19:40:03 +08:00
7e710701ce [enhancement](bakcup) throw detailed msg for backup or restore (#42288) (#42312)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-24 16:56:04 +08:00
8ff77c9509 [improve](restore) Split batch creating replica task by table id #42235 (#42342)
cherry pick from #42235
2024-10-24 16:20:19 +08:00
aa0fe8ccf4 [fix] (inverted index) fix the error result in the query when using count on index (#42346) (#42347)
## Proposed changes

pick from master #42346
2024-10-24 12:11:59 +08:00
5453337e7b [fix](mtmv)Fix the old version of the materialized view error 'Curren… (#42307)
…t database is not set'` (#42152)

pick: https://github.com/apache/doris/pull/42152
2024-10-24 11:16:01 +08:00
cbdaaa62b2 [feature](function) hour/minute/second functions support time as an a… 41008 (#42232)
…rgument. (#41008)

## Proposed changes

```
mysql> select cast(4562632 as time),hour(cast(4562632 as time)), minute(cast(4562632 as time)),second(cast(4562632 as time));
+-----------------------+-----------------------------+-------------------------------+-------------------------------+
| cast(4562632 as TIME) | hour(cast(4562632 as TIME)) | minute(cast(4562632 as TIME)) | second(cast(4562632 as TIME)) |
+-----------------------+-----------------------------+-------------------------------+-------------------------------+
| 456:26:32             |                         456 |                            26 |                            32 |
+-----------------------+-----------------------------+-------------------------------+-------------------------------+
```

<!--Describe your changes.-->

---------

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: Dongyang Li <hello_stephen@qq.com>
2024-10-24 11:09:36 +08:00
73fc5c734e [fix](mtmv)fix in the scenario of recreating a table, the materialize… (#42339)
…d view may assume that the data has not changed (#41762)

pick: https://github.com/apache/doris/pull/41762
2024-10-24 11:07:29 +08:00