Commit Graph

9279 Commits

Author SHA1 Message Date
01acd60e2b [Pick][Improment]Add workload group query detail (#40050) (#41594)
pick #40050
2024-10-11 09:32:33 +08:00
0fb42d3a48 [Enhancement](tvf)catalog tvf implements user permission checks and hides sensitive information (#41497) (#41604)
bp #41497 

before #21790
## Proposed changes
This PR unifies the duplicate parts of `catalog tvf` and `show
catalogs`, adds permission check when querying `catalog tvf`, and hides
sensitive information.
2024-10-10 17:55:40 +08:00
3120bfb6e3 [fix](pipelinex) fix fragment instance progress reports (part 2) (#40694) (#41641)
backport #40694
2024-10-10 17:49:41 +08:00
d32688e091 [Enhancement](multi-catalog) Set hdfs native client logger to glog and redirect jvm stdout/stderr logger to jni.log. (#41633)
Backport #39540.

Co-authored-by: Mingyu Chen <morningman@163.com>
2024-10-10 17:47:21 +08:00
070bd6cf99 [fix](jdbc) fix Unknown command(27) (#41621) (#41637)
pick https://github.com/apache/doris/pull/41621
2024-10-10 17:46:45 +08:00
cd26713c26 [fix](Nereids) could not parse date/datetime with blank + zone (#41394) (#41620)
pick from master #41394

for example:
2008-08-08 20:08:08 +08:00 parse failed because the blank before +08:00
2024-10-10 14:03:41 +08:00
33fad04341 [opt](Nereids) use 1 instead narrowest column when do column pruning (#41548) (#41627)
pick from master #41548
2024-10-10 14:02:23 +08:00
eb348ec10e [improve](binlog) Filter the truncated partitions #41611 (#41630)
cherry pick from #41611
2024-10-10 11:59:27 +08:00
6a3dc98361 [fix](Nereids) merge request properties map of group should compare cost (#40819) (#41615)
pick from master #40819

when do merge, we should update target requestPropertiesMap ONLY IF the
cost of source's request property lower than target one. Otherwise, the
requestPropertiesMap will not sync with lowestCostTable. Then, we will
get wrong output property when get the final plan.
2024-10-10 11:16:00 +08:00
032c00e3c9 [branch-2.1](create table) show failed detail msg #41463 (#41544)
cherry-pick: #41463
2024-10-10 11:15:02 +08:00
a45dc8796a [fix](Nereids) simplify decimal comparison wrong when cast to smaller scale (#41151) (#41618)
pick from master #41151
2024-10-09 23:03:01 +08:00
649cefd70f [opt](Nereids) forbid distribute under project and filter (#39812) (#41622)
pick from master #39812
2024-10-09 23:02:06 +08:00
1d2e2adb74 [fix](parser) should not use selectHint in any place (#41260) (#41619)
pick from master #41260

because all comment has been redirect to channel 2, we should not use
hint in any place. selectHint only use to parse hint.
2024-10-09 23:00:04 +08:00
ade86c0600 Revert "[branch-2.1][improvement](jdbc catalog) Optimize JdbcCatalog case mapping stability" (#41588)
Reverts apache/doris#41330
2024-10-09 20:51:58 +08:00
b6c0603e0d [fix](mtmv)Fix show mtmv time should wrapped in double quotation mark… (#41501)
pick: https://github.com/apache/doris/pull/41419
2024-10-09 12:19:28 +08:00
25684f487b [2.1][improvement](jdbc catalog) Improve JdbcClientException to accommodate various identifier formats (#41530)
pick (#40931)

In some cases, JDBC returns exceptions with various identifiers that
cannot be formatted correctly, such as `%`. This PR optimizes this.
2024-10-09 10:32:41 +08:00
a0aed77218 [cherry-pick](branch2.1) fix hudi jni scanner (#41566)
pick from https://github.com/apache/doris/pull/41316
2024-10-09 10:31:50 +08:00
ca59c8537c [chore](query err) fix cutting err msg incorrectly #41456 (#41538)
cherry pick from #41456
2024-10-08 16:09:15 +08:00
3355ebe5e7 [enhancement](delete) Using insert timeout session var to control delete job timeout (#41063) (#41475)
## Proposed changes

pick: #41063
2024-10-08 12:03:30 +08:00
806fe98a93 [2.1][fix](oracle scan) Fix performance issues caused by version judgment (#41509)
pick (#41407)
2024-10-01 21:04:22 +08:00
4f81fc474c [bugfix](paimon)Get the file format by file name (#41020) (#41487)
bp #41020
2024-09-30 15:46:13 +08:00
b5e94b65c0 [fix](mtmv) Generate mtmv cache should use ADMIN user, and rewritten plan should not check privilege (#40374) (#41450)
## Proposed changes

pr: https://github.com/apache/doris/pull/40374
commitId: f3d92e48
2024-09-30 09:59:54 +08:00
a7b88af861 [bugfix](external)Memory leak problem for external table with insert operation for 2.1 (#40440) (#41479)
bp: #40440
2024-09-29 23:09:25 +08:00
9864324286 [enhancement](err-msg) Clarify error msg for forward queries (#40802) (#41474)
## Proposed changes

pick: #40802
2024-09-29 22:23:13 +08:00
d659750fd9 [pick](Serde-2.1) fix variant serde may lost num_rows when subcolumns empty (#41438)
serialization object with empty subcolumns may lost num_rows, so need to
record num_rows and set back num_rows in serdes

backport #38413
2024-09-29 09:45:37 +08:00
9a9226e541 [fix](block_rule) SQL block rule not working after FE restart (#41228) (#41250)
pick: https://github.com/apache/doris/pull/41228
2024-09-28 10:08:59 +08:00
514184ee12 [fix](audit_loader) fix that old external audit loader plugin not work because of incompatibility with new audit plugin (#40565) (#41400)
bp #40565

Co-authored-by: caiconghui <55968745+caiconghui@users.noreply.github.com>
Co-authored-by: caiconghui1 <caiconghui1@jd.com>
2024-09-28 08:38:14 +08:00
afd87e9aee [fix](nereids)should prune logicalSink's child node in ColumnPruning (#41289) (#41424)
## Proposed changes

pick from master https://github.com/apache/doris/pull/41289

<!--Describe your changes.-->
2024-09-28 08:35:35 +08:00
6dd94c6bda [opt](log) add warn log for saving and pushing image failure (#41216) (#41434)
bp #41216
2024-09-28 08:34:45 +08:00
1aea05b5b0 [improvement](statistics)Support partition row count return -1 when it is not fully reported. (#41348) (#41408)
backport: https://github.com/apache/doris/pull/41348
2024-09-27 22:17:16 +08:00
0b4552f74b [cherry-pick](branch-2.1) pick hive text write from master (#40537)
## Proposed changes
pick prs:
https://github.com/apache/doris/pull/38549
https://github.com/apache/doris/pull/40183
https://github.com/apache/doris/pull/40315

---------

Co-authored-by: Calvin Kirs <kirs@apache.org>
2024-09-27 20:57:07 +08:00
82228358b9 [Fix](nereids) fix create view with nullable column (#41234) (#41393)
cherry-pick from master #41234
2024-09-27 19:13:54 +08:00
0c51ee26ea [fix](function) add time type in conditional-functions (#41270) (#41379)
## Proposed changes
https://github.com/apache/doris/pull/41270

<!--Describe your changes.-->
2024-09-27 17:19:54 +08:00
1baf0dbc59 [improvement](statistics)Reduce partition column sample BE memory consumption. (#41203) (#41387)
backport: https://github.com/apache/doris/pull/41203
2024-09-27 13:46:24 +08:00
21b8887b54 [feat](profile) Print changed session var in profile #41016 (#41317)
cherry pick from #41016
2024-09-27 09:55:43 +08:00
c487fc0e5f [enhance](mtmv)Optimize the logic of mtmv lock (#41010) (#41254)
pick: https://github.com/apache/doris/pull/41010
2024-09-26 23:02:02 +08:00
eb13cd4154 [branch-2.1] Picks "[Fix](partial update) Fix __DORIS_SEQUENCE_COL__ is not set for newly inserted rows in partial update #40272" (#40964)
picks https://github.com/apache/doris/pull/40272
2024-09-26 22:54:27 +08:00
e6ce4a2c26 [fix](meta) fix Unknown column 'mva_SUM__CAST` (#41285)
## Proposed changes

cherry pick from #41283
2024-09-26 22:52:51 +08:00
3eda77b3d9 [branch-2.1][improvement](jdbc catalog) Optimize JdbcCatalog case mapping stability (#41330)
pick #40891
This PR makes the following changes to the uppercase and lowercase
mapping of JdbcCatalog
1. The identifierMapping is managed by JdbcExternalCatalog instead of
JdbcClient to better control its lifecycle
2. The identifierMapping no longer loads remoteName alone, but Catalog
controls the loading uniformly
3. The identifierMapping will be loaded when each FE performs
makeSureInitialized() to ensure that each FE has a mapping
4. The initialization of mapping will only be performed once in
makeSureInitialized(), which means that even if you use metaCache, if
your source data is updated when identifierMapping is enabled, you must
refresh the catalog to query normally.
5. The identifierMapping is only responsible for the properties of the
Catalog and is no longer affected by the fe config, simplifying the
processing logic
6. If lower_case_mete_names is false and meta_names_mapping is empty in
the catalog properties, the identifierMapping will no longer take
effect, further enhancing the stability of the default settings
7. The JdbcClient is no longer closed during onRefreshCache, reducing
the repeated creation of resources, improving reuse, and reducing the
leakage of some global shared threads
2024-09-26 22:49:55 +08:00
5e6cd8749e [branch-2.1](hudi) fix fe memory leak(#41256) (#41225)
bp #41256
2024-09-26 22:34:16 +08:00
d89e5de815 [improvement](tablet scheduler) fix higher priority tablet add failed due to pending queue full #41076 (#41268)
cherry pick from #41076
2024-09-26 22:31:20 +08:00
f422b82e24 [Fix](Nereids) fix append_trailing_char_if_absent function return null (#41157)
cherry-pick: https://github.com/apache/doris/pull/40820
example: select append_trailing_char_if_absent('it','a') would return
null in original design, it can not return null when folding constant on
fe any time
2024-09-26 16:18:12 +08:00
43c1066bc7 [fix](analysis) Fix ColumnDef to sql result #41205 (#41325)
cherry pick from #41205
2024-09-26 14:11:13 +08:00
4deda2fce7 [improvement](nereids) Simplify ScanNode projection handling by removing redundant conditions (#40801) (#41315)
pick from master #40801

This PR simplifies the handling of `ScanNode` projection logic.
Previously, the code included multiple conditional checks to determine
whether a `projectionTuple` should be generated. These conditions have
been removed, and now `projectionTuple `is always generated for
`ScanNode`, ensuring a consistent projection setup. Additionally,
redundant handling of `SlotId` and `SlotRef` has been eliminated, making
the code cleaner and easier to maintain. The behavior for `OlapScanNode`
remains unchanged.
2024-09-26 10:35:01 +08:00
1459517568 [improvement](binlog) filter dropped indexes #41246 (#41300)
cherry pick from #41246
2024-09-26 08:38:28 +08:00
306af52aa2 [Fix](jdbc-scanner) Fix jdbc scanner memory leak because it didn't close outputTable. (#41266)
## Proposed changes

Backport #41041.
2024-09-25 17:35:13 +08:00
63555ea970 [fix](mysql)Support COM_CHANGE_USER and other mysql command. (#40932) (#41251)
backport: https://github.com/apache/doris/pull/40932
2024-09-25 15:37:02 +08:00
ba491af9f1 [branch-2.1] do not fallbacl to origin planner for call stmt (#41263)
`CALL xxx` statement is not supported by old planner.
And if fallback, the error message will be confused.
2024-09-25 14:30:56 +08:00
5bf210ffa4 [revert](storage) storage medium of partition should not inherit from… (#41212)
… table (#41192)
pick: https://github.com/apache/doris/pull/41192
2024-09-25 14:15:24 +08:00
11bad4cbc9 [opt](routine load) optimize routine load timeout logic (#40818) (#41135)
pick #40818

If IO/CPU resources are tight, routine load task is likely to timeout.
The current method is self-adaption backoff
https://github.com/apache/doris/pull/32227, but the problem is it will
do some ineffective work to match proper timeout. For one routine load
task, a better way to handle task is finishing executing instead of
retry when resources are tight. Therefore, this pr increase timeout to
make "task always finish even if it is slow when resources are tight".
2024-09-25 14:14:02 +08:00