Commit Graph

20012 Commits

Author SHA1 Message Date
c50d2ea4e2 [fix](test) fix unstable test (#40143)
## Proposed changes

cherry pick from #40142
2024-08-29 23:42:24 +08:00
82fb747e6f [fix](Nereids) datetimev2 literal equals should compare microsecond (#40121) (#40150)
pick from master #40121
2024-08-29 23:41:51 +08:00
fe881f8676 [fix](core) Fix BUCKET_HASH_SHUFFLE local exchanger (#40149)
## Proposed changes

pick #40148

<!--Describe your changes.-->
2024-08-29 23:23:42 +08:00
090474a815 [branch-2.1](load) Lower lock granularity to reduce loading CPU pressure (#40135)
pick https://github.com/apache/doris/pull/40134
2024-08-29 23:22:58 +08:00
a7156ee775 [fix](parquet)Fix the be core issue when reading parquet unsigned types. (#39926) (#40123)
bp #39926
2024-08-29 21:52:52 +08:00
841565a40d [fix](local shuffle) Fix hash shuffle local exchanger (#40101)
## Proposed changes

pick #40036

<!--Describe your changes.-->
2024-08-29 16:52:28 +08:00
c6df7c21a3 [branch-2.1](hive) support hive write text table (#38549) (#40063)
1. Support write hive text table
2. Add SessionVariable `hive_text_compression` to write compressed hive
text table
3. Supported compression type: gzip, bzip2, snappy, lz4, zstd

pick from https://github.com/apache/doris/pull/38549
2024-08-29 16:50:40 +08:00
901344a880 [chore](log) Log written rows when result sink close #39647 (#40011)
cherry pick from #39647
2024-08-29 16:49:44 +08:00
2054cc9fc9 [fix](nereids) fix union all instance number (#40099)
pick from https://github.com/apache/doris/pull/39999

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
2024-08-29 16:47:55 +08:00
ac8cff34ce [fix](ut)fix be enable_http_auth ut (#40071) (#40088)
bp #40071 

## Proposed changes
before pr #39577
2024-08-29 16:42:53 +08:00
51e8d9174d [fix](compaction) fix heap-use-after-free caused by compaction sample info resize (#40058) (#40077) 2024-08-29 16:42:35 +08:00
eb6a040a03 [branch-2.1] Picks "[Fix](partial update) Fix wrongly update autoinc column in partial update #39996" (#40073)
## Proposed changes

pick https://github.com/apache/doris/pull/39996
2024-08-29 16:41:51 +08:00
131238ff71 [fix](file-cache) change metric_value column in file_cache_statistics table to string (#40083)
Make it more flexible
followup #39552
2024-08-29 16:39:22 +08:00
8df93f8dfe [Opt](parquet/orc-reader) Opt get dict ids in _rewrite_dict_predicates() (#40095)
## Proposed changes

backport #39893.
2024-08-29 14:50:42 +08:00
15b14ef49b [fix](inverted index) fix error handling in fast_execute (#40086)
https://github.com/apache/doris/pull/40024
2024-08-29 14:45:25 +08:00
7155711431 [cherry-pick](branch-2.1) Improve local shuffle strategy (#40030)
pick #34122 #35454 #35716 #37195
2024-08-29 14:16:16 +08:00
34d95cef1f [improve](ES Catalog)Only push down literal expr in binary predicate(#39845) (#39847)
## Proposed changes

bp #39845
2024-08-29 14:07:17 +08:00
c70d7d8423 [fix](sink) Do not block result sink on pipeline engine (#40094)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-29 14:04:25 +08:00
93da0ebaf4 [chore](backup) limit the involved tablets in a backup job #39987 (#40080)
cherry pick from #39987
2024-08-29 12:03:14 +08:00
c5d9e178be [fix](mtmv) Fix result wrong when query rewrite by mv if query contains null_unsafe equals expression (#39629) (#40041)
## Proposed changes

commitId: 5d4ad028
pr: https://github.com/apache/doris/pull/39629
2024-08-29 00:31:36 +08:00
c3c6a4fc4c [pick](Variant) variant fallthrough with inverted index (#40070)
backport #40069
2024-08-29 00:30:37 +08:00
bb709ad917 [branch-2.1] Picks "[Fix](merge-on-write) Fix duplicate key problem after adding sequence column for merge-on-write table #39958" (#40010)
## Proposed changes

picks https://github.com/apache/doris/pull/39958
2024-08-28 23:00:31 +08:00
688b97e183 [enhancement](Nereids) support reuse sql cache between different comment (#40065)
cherry pick from #40049
2024-08-28 22:15:55 +08:00
c9aca7d852 [opt](profile) Add ExecutedByFrontend in profile #39942 (#40012)
cherry pick from #39942
2024-08-28 22:06:57 +08:00
7ffbead040 [branch-2.1] Picks "[Fix](regression) fix unstable case test_schema_change_unique_mow #39315" (#40034)
## Proposed changes

picks https://github.com/apache/doris/pull/39315/files
2024-08-28 22:02:28 +08:00
af6c39d770 [fix](compile) compile fail on MacOS because 'bits/types/struct_timespec.h' not found (#39567) (#40044)
pick #39567 to branch-2.1

Co-authored-by: 神技圈子 <songguangfan@gmail.com>
Co-authored-by: 宋光璠 <songguangfan@sf.com>
2024-08-28 22:01:49 +08:00
1d439d2ea9 [opt](parquet) add predicate filter time for parquet reader (#40005) (#40053)
bp #40005
2024-08-28 21:57:00 +08:00
a32a2d8bb4 [bugfix](iceberg)Restrictions on creating a database (#39641) (#40051)
bp #39641

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-08-28 21:45:03 +08:00
d1c3b857cf [fix](catalog) fix invalid getPartitionNames() methods (#39878) (#40054)
bp #39878
2024-08-28 21:44:30 +08:00
6ef1211585 [bugfix](paimon)Use the fs cache, otherwise it will cause OOM (#38552) (#40052)
bp #38552

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-08-28 21:42:36 +08:00
4909c34555 [fix](compile) Fix Type Mismatch in min Function for VMysqlResultWriter on macOS (#38202) (#40042)
pick #38202 to branch-2.1

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2024-08-28 21:36:04 +08:00
b052f46944 [fix](json-quote) fix json quote func for not find the func (#39931) (#40061)
1.  fix function not found before this pr:
```
drop table if exists t003;
create table t003 (a bigint, b json not null) properties ("replication_num"="1");
insert into t003 values (1, '{"a":1,"b":2}');
select a, map_agg("k1", json_quote(b)) from t003 group by a;

[17:47]>  select a, map_agg("k1", json_quote(b)) from t003 group by a;
(1105, 'errCode = 2, detailMessage = (172.20.48.119)[INTERNAL_ERROR]Function json_quote get failed, expr is VectorizedFnCall[json_quote](arguments=(CAST b(JSONB) TO String),return=Nullable(String)) and return type is Nullable(String).')
``` 
after pr , we can carry on it 
2. fix a core if we use json_quote with table column
```
[WARNING!] /sys/kernel/mm/transparent_hugepage/enabled: [always] madvise never, Doris not recommend turning on THP, which may cause the BE process to use more memory and cannot be freed in time. Turn off THP: `echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled`
start BE in local mode
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
*** Query id: b7b94dead55e4090-8c8e94f65ec9efd3 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1724744881 (unix time) try "date -d @1724744881" if you are using GNU date ***
*** Current BE git commitID: bd5844ea0d ***
*** SIGABRT unknown detail explain (@0x461003611e1) received by PID 3543521 (TID 3548858 OR 0x7f8a53c1a700) from PID 3543521; stack trace: ***
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/wangqiannan/amory/doris/be/src/common/signal_handler.h:421
 1# 0x00007F9429595B50 in /lib64/libc.so.6
 2# gsignal in /lib64/libc.so.6
 3# __GI_abort in /lib64/libc.so.6
 4# _nl_load_domain.cold.0 in /lib64/libc.so.6
 5# 0x00007F942958E426 in /lib64/libc.so.6
 6# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::Prefix(rapidjson::Type) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488
 7# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::String(char const*, unsigned int, bool) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:206
 8# bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u> >(rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>&) const at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/document.h:1974
 9# doris::vectorized::FunctionJsonQuoteImpl::execute(std::vector<doris::vectorized::ColumnStr<unsigned int> const*, std::allocator<doris::vectorized::ColumnStr<unsigned int> const*> > const&, doris::vectorized::ColumnStr<unsigned int>&, unsigned long) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:826
10# doris::vectorized::FunctionJson<doris::vectorized::FunctionJsonQuoteImpl>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:985
11# doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:463
12# doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:120
13# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:245
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-28 21:31:08 +08:00
799ad1d398 [fix](mv) Fix test create table like test unstable (#39305) (#40013)
ddl_p0/test_create_table_like_nereids.groovy test result is unstable.
Should wait util rollup join finished then we can compare the explain
rollup result.

## Proposed changes

commitId: 3b7b29b8
pr: https://github.com/apache/doris/pull/39305
2024-08-28 18:42:35 +08:00
fa4d885487 [fix](regression) fix compaction_width_array_column case (#37664) (#40045)
pick master #37664
2024-08-28 18:13:42 +08:00
b3b085c561 [fix](statistics)Use utf-8 charset for internal query result. (#39989) (#40026)
backport: https://github.com/apache/doris/pull/39989
2024-08-28 16:24:34 +08:00
42f30d4a48 [branch-2.1][Bug] test suite test_table_options may cause NPE in FE #39457 (#39509) (#40018)
## Proposed changes

pick #39509

Co-authored-by: Vallish Pai <vallishpai@gmail.com>
2024-08-28 14:17:00 +08:00
ddf72ce09a [Fix](branch-2.1) fix manual pick with mistake when handling decimal … (#40008)
…type

introduced by #39843
2024-08-28 10:15:52 +08:00
6915d76731 [opt](file-cache) add evict file number per round (#39721)
Previously, when getting block from file cache, it may try to evict
lots of blocks to reserve capacity for lru cache. This operation may
take long time
while hold the lock, causing other operation blocked.

This PR add a new BE config `file_cache_max_evict_num_per_round`,
default is 1000, so that it will not hold lock for a long time.
2024-08-28 08:49:12 +08:00
cb312cabb2 [Fix](tablet-meta) limit the data size of tablet meta (#39455) (#39974)
pick master #39455
2024-08-27 20:10:17 +08:00
29088ebaed [enhancement](compression) prints the specific exception when ZSTD compress fails (#39433) (#39978)
cherry-pick #39433 to branch-2.1
2024-08-27 20:07:15 +08:00
34d80f6239 [fix](group commit) Fix group commit error log when decommission (#39899) (#39955)
pick https://github.com/apache/doris/pull/39899
2024-08-27 18:33:24 +08:00
4127eec9a7 [fix](function) fix error return type in mod(float32,BigInt) (#39358) (#39971)
## Proposed changes
https://github.com/apache/doris/pull/39358
```
CREATE TABLE testdb (
    K1 BIGINT,
    K2 FLOAT
) properties("replication_num" = "1");
insert into testdb values(1,1.1);
select mod(k1,k2) from testdb;
mysql [test10]>select mod(k1,k2) from testdb;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Function mod get failed, expr is VectorizedFnCall[mod](arguments=K1, K2,return=Nullable(Float32)) and return type is Nullable(Float32).
```

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-27 18:32:52 +08:00
d70fe130d7 [fix](explode-map) fix explode_map with alias (#39972)
## Proposed changes
backport: https://github.com/apache/doris/pull/39757
Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-27 18:32:10 +08:00
fc9936d923 [fix](funcs) fix map struct construct funcs (#39973)
## Proposed changes
backport: https://github.com/apache/doris/pull/39699
Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-27 18:31:51 +08:00
da292b6908 [Fix-2.1](testcase) fix an unstable semi-structure case result (#39981)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-27 18:31:32 +08:00
173aafc86f [Enhancement] add information_schema.table_properties #38745 (#38746) (#39886)
bp #38746

---------

Co-authored-by: Vallish Pai <vallishpai@gmail.com>
2024-08-27 17:22:19 +08:00
bbf7701046 [fix](catalog) hide some properties in show create catalog stmt (#39946) (#39970)
bp #39946
2024-08-27 16:46:12 +08:00
06a2182d89 [fix](build index)Only get file size for inverted index (#39966)
## Proposed changes

Fix `NOT FOUND` index file error message in logs.
bp #39965
2024-08-27 15:30:51 +08:00
bf7675dc94 [Pick](PreparedStatement) allow prepare mysql command in Nerids even … (#39960)
…if enableServeSidePreparedStatement is false to avoid fallback to
legacy planner

backport #39465
2024-08-27 14:58:28 +08:00
b7e0bfa1c2 [improve](function) opt aes_encrypt/decrypt function to handle const column (#37194) (#39954)
pick #37194 to branch-2.1

---------

Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com>
2024-08-27 14:56:38 +08:00