Commit Graph

7548 Commits

Author SHA1 Message Date
1ecc85b575 [Fix](ctas) forward the use_max_length_of_varchar_in_ctas (#39705) (#40138)
cherry-pick from master #39705
2024-08-30 10:11:29 +08:00
0b9817ea4b [cherry-pick](branch-2.1) Pick "[Enhancement](txn) Block new insert into if schema change happens during transaction (#39483)" (#40115)
## Proposed changes

Pick #39483 

<!--Describe your changes.-->
2024-08-30 10:01:10 +08:00
45e35e8a99 [improvement](mtmv) Optimize get available mv logic to avoid unnecessary rewrite (#39734) (#40136)
## Proposed changes

commitId: 55ed1277
pr: https://github.com/apache/doris/pull/39734
2024-08-30 08:02:27 +08:00
e072ce7f68 [fix](nereids) Remove Nondeterministic to avoid ambiguity when decide a expression is nondeterministic or not (#39801) (#40130)
## Proposed changes

commitId: 2b715a7e
pr: https://github.com/apache/doris/pull/39801
2024-08-30 07:51:42 +08:00
b86e4e8498 [fix](profile) fix possible coredump of rpc verbose profile (#40117)
## Proposed changes

Issue Number: close #xxx

`_instance_to_rpc_stats_vec` may be updated when sorting in
`ExchangeSinkBuffer<Parent>::update_profile`, which may cause coredump.
2024-08-29 23:54:52 +08:00
82fb747e6f [fix](Nereids) datetimev2 literal equals should compare microsecond (#40121) (#40150)
pick from master #40121
2024-08-29 23:41:51 +08:00
c6df7c21a3 [branch-2.1](hive) support hive write text table (#38549) (#40063)
1. Support write hive text table
2. Add SessionVariable `hive_text_compression` to write compressed hive
text table
3. Supported compression type: gzip, bzip2, snappy, lz4, zstd

pick from https://github.com/apache/doris/pull/38549
2024-08-29 16:50:40 +08:00
2054cc9fc9 [fix](nereids) fix union all instance number (#40099)
pick from https://github.com/apache/doris/pull/39999

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
2024-08-29 16:47:55 +08:00
eb6a040a03 [branch-2.1] Picks "[Fix](partial update) Fix wrongly update autoinc column in partial update #39996" (#40073)
## Proposed changes

pick https://github.com/apache/doris/pull/39996
2024-08-29 16:41:51 +08:00
131238ff71 [fix](file-cache) change metric_value column in file_cache_statistics table to string (#40083)
Make it more flexible
followup #39552
2024-08-29 16:39:22 +08:00
7155711431 [cherry-pick](branch-2.1) Improve local shuffle strategy (#40030)
pick #34122 #35454 #35716 #37195
2024-08-29 14:16:16 +08:00
34d95cef1f [improve](ES Catalog)Only push down literal expr in binary predicate(#39845) (#39847)
## Proposed changes

bp #39845
2024-08-29 14:07:17 +08:00
93da0ebaf4 [chore](backup) limit the involved tablets in a backup job #39987 (#40080)
cherry pick from #39987
2024-08-29 12:03:14 +08:00
c5d9e178be [fix](mtmv) Fix result wrong when query rewrite by mv if query contains null_unsafe equals expression (#39629) (#40041)
## Proposed changes

commitId: 5d4ad028
pr: https://github.com/apache/doris/pull/39629
2024-08-29 00:31:36 +08:00
688b97e183 [enhancement](Nereids) support reuse sql cache between different comment (#40065)
cherry pick from #40049
2024-08-28 22:15:55 +08:00
c9aca7d852 [opt](profile) Add ExecutedByFrontend in profile #39942 (#40012)
cherry pick from #39942
2024-08-28 22:06:57 +08:00
a32a2d8bb4 [bugfix](iceberg)Restrictions on creating a database (#39641) (#40051)
bp #39641

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-08-28 21:45:03 +08:00
d1c3b857cf [fix](catalog) fix invalid getPartitionNames() methods (#39878) (#40054)
bp #39878
2024-08-28 21:44:30 +08:00
6ef1211585 [bugfix](paimon)Use the fs cache, otherwise it will cause OOM (#38552) (#40052)
bp #38552

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-08-28 21:42:36 +08:00
b052f46944 [fix](json-quote) fix json quote func for not find the func (#39931) (#40061)
1.  fix function not found before this pr:
```
drop table if exists t003;
create table t003 (a bigint, b json not null) properties ("replication_num"="1");
insert into t003 values (1, '{"a":1,"b":2}');
select a, map_agg("k1", json_quote(b)) from t003 group by a;

[17:47]>  select a, map_agg("k1", json_quote(b)) from t003 group by a;
(1105, 'errCode = 2, detailMessage = (172.20.48.119)[INTERNAL_ERROR]Function json_quote get failed, expr is VectorizedFnCall[json_quote](arguments=(CAST b(JSONB) TO String),return=Nullable(String)) and return type is Nullable(String).')
``` 
after pr , we can carry on it 
2. fix a core if we use json_quote with table column
```
[WARNING!] /sys/kernel/mm/transparent_hugepage/enabled: [always] madvise never, Doris not recommend turning on THP, which may cause the BE process to use more memory and cannot be freed in time. Turn off THP: `echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled`
start BE in local mode
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
*** Query id: b7b94dead55e4090-8c8e94f65ec9efd3 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1724744881 (unix time) try "date -d @1724744881" if you are using GNU date ***
*** Current BE git commitID: bd5844ea0d ***
*** SIGABRT unknown detail explain (@0x461003611e1) received by PID 3543521 (TID 3548858 OR 0x7f8a53c1a700) from PID 3543521; stack trace: ***
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/wangqiannan/amory/doris/be/src/common/signal_handler.h:421
 1# 0x00007F9429595B50 in /lib64/libc.so.6
 2# gsignal in /lib64/libc.so.6
 3# __GI_abort in /lib64/libc.so.6
 4# _nl_load_domain.cold.0 in /lib64/libc.so.6
 5# 0x00007F942958E426 in /lib64/libc.so.6
 6# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::Prefix(rapidjson::Type) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488
 7# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::String(char const*, unsigned int, bool) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:206
 8# bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u> >(rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>&) const at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/document.h:1974
 9# doris::vectorized::FunctionJsonQuoteImpl::execute(std::vector<doris::vectorized::ColumnStr<unsigned int> const*, std::allocator<doris::vectorized::ColumnStr<unsigned int> const*> > const&, doris::vectorized::ColumnStr<unsigned int>&, unsigned long) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:826
10# doris::vectorized::FunctionJson<doris::vectorized::FunctionJsonQuoteImpl>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:985
11# doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:463
12# doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:120
13# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:245
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-28 21:31:08 +08:00
b3b085c561 [fix](statistics)Use utf-8 charset for internal query result. (#39989) (#40026)
backport: https://github.com/apache/doris/pull/39989
2024-08-28 16:24:34 +08:00
42f30d4a48 [branch-2.1][Bug] test suite test_table_options may cause NPE in FE #39457 (#39509) (#40018)
## Proposed changes

pick #39509

Co-authored-by: Vallish Pai <vallishpai@gmail.com>
2024-08-28 14:17:00 +08:00
34d80f6239 [fix](group commit) Fix group commit error log when decommission (#39899) (#39955)
pick https://github.com/apache/doris/pull/39899
2024-08-27 18:33:24 +08:00
d70fe130d7 [fix](explode-map) fix explode_map with alias (#39972)
## Proposed changes
backport: https://github.com/apache/doris/pull/39757
Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-27 18:32:10 +08:00
173aafc86f [Enhancement] add information_schema.table_properties #38745 (#38746) (#39886)
bp #38746

---------

Co-authored-by: Vallish Pai <vallishpai@gmail.com>
2024-08-27 17:22:19 +08:00
bbf7701046 [fix](catalog) hide some properties in show create catalog stmt (#39946) (#39970)
bp #39946
2024-08-27 16:46:12 +08:00
bf7675dc94 [Pick](PreparedStatement) allow prepare mysql command in Nerids even … (#39960)
…if enableServeSidePreparedStatement is false to avoid fallback to
legacy planner

backport #39465
2024-08-27 14:58:28 +08:00
4cf769b39f [Improve](table-function) explode json array with json args (#39491) 2024-08-27 14:53:17 +08:00
a4c5c03acf [fix](mtmv) Add debug log decide for performance when query rewrite by materialized view (#39914) (#39952)
## Proposed changes
In method `AbstractMaterializedViewRule#isMaterializationValid` Should
add `LOG.isDebugEnabled()` before print debug log. Because
`Plan#treeString` in debug log is performance consume.

## Proposed changes

pr: https://github.com/apache/doris/pull/39914 
commitId: 45ddb8ce
2024-08-27 14:27:44 +08:00
959ea278cb [fix](replay) fix replay ReplicaPersistInfo no update lastFailedVersion and lastSuccVersion #39918 (#39947)
cherry pick from #39918
2024-08-27 11:58:13 +08:00
db0724dfe0 [Fix-2.1](function) fix function covar core for not null input (#39943)
## Proposed changes

Issue Number: close #xxx

add testcases like:
```groovy
    qt_notnull1 "select covar_samp(non_nullable(x), non_nullable(y)) from test_covar_samp"
    qt_notnull2 "select covar_samp(x, non_nullable(y)) from test_covar_samp"
    qt_notnull3 "select covar_samp(non_nullable(x), y) from test_covar_samp"
```

before they will all coredump in 2.1
2024-08-27 08:39:47 +08:00
aacd61a98f [fix](nereids)filter estimation for slot=unknown #39592 (#39933)
## Proposed changes
pick #39592
Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-27 08:03:00 +08:00
e4bf10ee96 [enhancement](Load)allow load data to the other partitions when some partitions are restoring (#39915)
If broker load or stream load task execute in one table that is
restoring data, load task will failed with Exception.
Exception info :"Table [xxx] is under restore" or "Table [xxx] is in
restore process, can't load into it".

But mostly restoreJob only effects some partitions in this table, not
all of them, so that the other partitions still need to load data
successfully.
To achieve this goal, before checking olap table state, check partition
state first.

cherry pick from master branch, pr has been merged:
https://github.com/apache/doris/pull/39595

Co-authored-by: shenshoucheng <shenshoucheng@jd.com>
2024-08-26 21:04:34 +08:00
fb5002d723 [fix](binlog) Fix NPE when recover binlogs (#39909) (#39910)
Cherry-pick #39909 

The field partition ID of DropPartitionInfo was added in PR:
apache/doris#37196, the old version doesn't contain this field so it
will be null.
2024-08-26 20:17:26 +08:00
9412ba56ae [fix](restore) Reset the db name of the materialized index stmt #39710 (#39848)
cherry pick from #39710, #39855
2024-08-26 16:51:50 +08:00
e21d2b82d6 [chore](backup) Fix the db name of the restored view (#38072)
Cherry-pick #37412, #39855

Previously, during restore, the database name in the CREATE VIEW
statement was not modified, causing the restored view to be unviewable
with the SHOW VIEW command. This PR retains the original cluster's
database name in the BackupMeta and manually replaces it with the new
cluster's database name in the CREATE VIEW statement during restore.
2024-08-26 14:12:47 +08:00
d87a220d2b [opt](privilege) Grant check name (#39597) (#39856)
pick https://github.com/apache/doris/pull/39597
2024-08-26 09:53:45 +08:00
9997911ec9 [feat](Nereids) support nereids hint position detaction (#39113) (#39417)
cherry-pick: #39113
When use hint in wrong position or use unsupport hint, use channel(2) to
filter it out

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-24 23:59:54 +08:00
8e140727ae [fix](udf) Fix exception thrown when replayDropFunction (#39868)
This PR completes #25965 to fix uncaught runtime exception when calling
`org.apache.doris.catalog.GlobalFunctionMgr#replayDropFunction`


![img_v3_02e0_3889b1de-583e-48ba-8e7f-48e1d3ae799g](https://github.com/user-attachments/assets/b0dbfefb-16f3-4dbb-8a00-f4773978ba01)
2024-08-24 23:56:48 +08:00
2dea859bdb [debug](rpc) debug rpc time consumption problem (#39852)
## Proposed changes

Issue Number: close #xxx

Add detail RPC time info for each channel, sorted by max rpc time of
channels:
```
                     DATA_STREAM_SINK_OPERATOR  (id=1,dst_id=1):
                          -  Partitioner:  Crc32HashPartitioner(64)
                          -  BlocksProduced:  74
                          -  BrpcSendTime:  2.689us
                          -  BrpcSendTime.Wait:  0ns
                          -  BytesSent:  89.35  KB
                          -  CloseTime:  680.152us
                          -  CompressTime:  0ns
                          -  ExecTime:  160.663ms
                          -  InitTime:  263.608us
                          -  InputRows:  32.512K  (32512)
                          -  LocalBytesSent:  0.00  
                          -  LocalSendTime:  0ns
                          -  LocalSentRows:  0
                          -  MemoryUsage:  
                              -  PeakMemoryUsage:  80.00  KB
                          -  MergeBlockTime:  0ns
                          -  OpenTime:  4.113ms
                          -  OverallThroughput:  0.0  /sec
                          -  PendingFinishDependency:  41.179ms
                          -  RowsProduced:  32.512K  (32512)
                          -  RpcAvgTime:  11.850ms
                          -  RpcCount:  10
                          -  RpcMaxTime:  86.891ms
                          -  RpcMinTime:  15.200ms
                          -  RpcSumTime:  118.503ms
                          -  SerializeBatchTime:  13.517ms
                          -  SplitBlockDistributeByChannelTime:  38.923ms
                          -  SplitBlockHashComputeTime:  2.659ms
                          -  UncompressedRowBatchSize:  135.19  KB
                          -  WaitForDependencyTime:  0ns
                              -  WaitForRpcBufferQueue:  0ns
                        RpcInstanceDetails:
                              -  Instance  85d4f75b72a9ea61:  Count:  4,  MaxTime:  36.238ms,  MinTime:  12.107ms,  AvgTime:  21.722ms,  SumTime:  86.891ms
                              -  Instance  85d4f75b72a9ea91:  Count:  3,  MaxTime:  11.107ms,  MinTime:  2.431ms,  AvgTime:  5.470ms,  SumTime:  16.412ms
                              -  Instance  85d4f75b72a9eac1:  Count:  3,  MaxTime:  7.554ms,  MinTime:  3.160ms,  AvgTime:  5.066ms,  SumTime:  15.200m
```
2024-08-24 19:59:39 +08:00
14a2a66106 [fix](paimon) fix not able to read paimon data from hdfs with HA (#39806) (#39876)
bp #39806
2024-08-24 17:51:15 +08:00
460605ae3c [branch-2.1] pick some prs (#39860)
## Proposed changes

Issue Number: close #xxx

https://github.com/apache/doris/pull/38385 optimize parsing datetime
https://github.com/apache/doris/pull/38978 make stream load failure
message more clear and disable some error's stacktrace by default
https://github.com/apache/doris/pull/39255 fix random function coredump
https://github.com/apache/doris/pull/39324 fix function corr
inconsistency with doc
https://github.com/apache/doris/pull/39449 check auto partitoin nullity
when creating partition
https://github.com/apache/doris/pull/39695 make
DynamicPartitionScheduler immediately know interval's change
https://github.com/apache/doris/pull/39754 Add some partition expr check
on creating table
2024-08-24 17:26:42 +08:00
9640e2de44 [fix](catalog) refactor location path and support default fs #39116 (#39203) 2024-08-24 16:05:13 +08:00
de2e8f0ae6 [fix](ctas) fix NPE when ctas with old planner and varchar issue (#39744) (#39871)
bp #39744
2024-08-24 09:24:47 +08:00
b9da934b16 [fix](hive) report error with escape char and null format (#39700) (#39869)
bp #39700

Co-authored-by: Socrates <suxiaogang223@icloud.com>
2024-08-24 09:23:03 +08:00
00e4f343b0 [enhancement](delete) Add a hint msg for forbidden delete when MV or rollup exists (#39505) (#39857)
## Proposed changes

When MV or Rollup exists, delete is forbidden on the base table
currently. Add a hint msg to indicate it.
2024-08-23 23:42:26 +08:00
9d597bde68 [fix](Nereids) producer to consumer should be multimap in cte (#39850) (#39867) 2024-08-23 23:25:11 +08:00
508c7a7040 [fix](hive)Modify the Hive notification event processing method when using meta cache and add parameters to the Hive catalog. (#39239) (#39865)
bp #39239

Co-authored-by: daidai <2017501503@qq.com>
2024-08-23 23:21:02 +08:00
19c51cc217 [fix](mtmv) use name instead of id in meta of MTMV (#39355) (#39748)
pick from master #39355
2024-08-23 20:40:54 +08:00
6ceb574aa0 [branch-2.1]Pick IO limit/workload group usage table (#39839) 2024-08-23 18:51:47 +08:00