database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,986 Commits 1 Branch 0 Tags
1d439d2ea9f9beec2c06bb4c97da89b23501e792
Commit Graph

19986 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Mingyu Chen
1d439d2ea9 [opt](parquet) add predicate filter time for parquet reader (#40005) (#40053) 
bp #40005
 2024-08-28 21:57:00 +08:00
Mingyu Chen
a32a2d8bb4 [bugfix](iceberg)Restrictions on creating a database (#39641) (#40051) 
bp #39641

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
 2024-08-28 21:45:03 +08:00
Mingyu Chen
d1c3b857cf [fix](catalog) fix invalid getPartitionNames() methods (#39878) (#40054) 
bp #39878
 2024-08-28 21:44:30 +08:00
Mingyu Chen
6ef1211585 [bugfix](paimon)Use the fs cache, otherwise it will cause OOM (#38552) (#40052) 
bp #38552

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
 2024-08-28 21:42:36 +08:00
camby
4909c34555 [fix](compile) Fix Type Mismatch in min Function for VMysqlResultWriter on macOS (#38202) (#40042) 
pick #38202 to branch-2.1

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
 2024-08-28 21:36:04 +08:00
amory
b052f46944 [fix](json-quote) fix json quote func for not find the func (#39931) (#40061) 
1.  fix function not found before this pr:
```
drop table if exists t003;
create table t003 (a bigint, b json not null) properties ("replication_num"="1");
insert into t003 values (1, '{"a":1,"b":2}');
select a, map_agg("k1", json_quote(b)) from t003 group by a;

[17:47]>  select a, map_agg("k1", json_quote(b)) from t003 group by a;
(1105, 'errCode = 2, detailMessage = (172.20.48.119)[INTERNAL_ERROR]Function json_quote get failed, expr is VectorizedFnCall[json_quote](arguments=(CAST b(JSONB) TO String),return=Nullable(String)) and return type is Nullable(String).')
``` 
after pr , we can carry on it 
2. fix a core if we use json_quote with table column
```
[WARNING!] /sys/kernel/mm/transparent_hugepage/enabled: [always] madvise never, Doris not recommend turning on THP, which may cause the BE process to use more memory and cannot be freed in time. Turn off THP: `echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled`
start BE in local mode
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488：void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
*** Query id: b7b94dead55e4090-8c8e94f65ec9efd3 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1724744881 (unix time) try "date -d @1724744881" if you are using GNU date ***
*** Current BE git commitID: bd5844ea0d ***
*** SIGABRT unknown detail explain (@0x461003611e1) received by PID 3543521 (TID 3548858 OR 0x7f8a53c1a700) from PID 3543521; stack trace: ***
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488：void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488：void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488：void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488：void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488：void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488：void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/wangqiannan/amory/doris/be/src/common/signal_handler.h:421
 1# 0x00007F9429595B50 in /lib64/libc.so.6
 2# gsignal in /lib64/libc.so.6
 3# __GI_abort in /lib64/libc.so.6
 4# _nl_load_domain.cold.0 in /lib64/libc.so.6
 5# 0x00007F942958E426 in /lib64/libc.so.6
 6# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::Prefix(rapidjson::Type) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488
 7# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::String(char const*, unsigned int, bool) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:206
 8# bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u> >(rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>&) const at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/document.h:1974
 9# doris::vectorized::FunctionJsonQuoteImpl::execute(std::vector<doris::vectorized::ColumnStr<unsigned int> const*, std::allocator<doris::vectorized::ColumnStr<unsigned int> const*> > const&, doris::vectorized::ColumnStr<unsigned int>&, unsigned long) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:826
10# doris::vectorized::FunctionJson<doris::vectorized::FunctionJsonQuoteImpl>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:985
11# doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:463
12# doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:120
13# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:245
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-28 21:31:08 +08:00
seawinde
799ad1d398 [fix](mv) Fix test create table like test unstable (#39305) (#40013) 
ddl_p0/test_create_table_like_nereids.groovy test result is unstable.
Should wait util rollup join finished then we can compare the explain
rollup result.

## Proposed changes

commitId: 3b7b29b8
pr: https://github.com/apache/doris/pull/39305
 2024-08-28 18:42:35 +08:00
Luwei
fa4d885487 [fix](regression) fix compaction_width_array_column case (#37664) (#40045) 
pick master #37664
 2024-08-28 18:13:42 +08:00
Jibing-Li
b3b085c561 [fix](statistics)Use utf-8 charset for internal query result. (#39989) (#40026) 
backport: https://github.com/apache/doris/pull/39989
 2024-08-28 16:24:34 +08:00
wangbo
42f30d4a48 [branch-2.1][Bug] test suite test_table_options may cause NPE in FE #39457 (#39509) (#40018) 
## Proposed changes

pick #39509

Co-authored-by: Vallish Pai <vallishpai@gmail.com>
 2024-08-28 14:17:00 +08:00
lihangyu
ddf72ce09a [Fix](branch-2.1) fix manual pick with mistake when handling decimal … (#40008) 
…type

introduced by #39843
 2024-08-28 10:15:52 +08:00
Mingyu Chen
6915d76731 [opt](file-cache) add evict file number per round (#39721) 
Previously, when getting block from file cache, it may try to evict
lots of blocks to reserve capacity for lru cache. This operation may
take long time
while hold the lock, causing other operation blocked.

This PR add a new BE config `file_cache_max_evict_num_per_round`,
default is 1000, so that it will not hold lock for a long time.
 2024-08-28 08:49:12 +08:00
Luwei
cb312cabb2 [Fix](tablet-meta) limit the data size of tablet meta (#39455) (#39974) 
pick master #39455
 2024-08-27 20:10:17 +08:00
Luwei
29088ebaed [enhancement](compression) prints the specific exception when ZSTD compress fails (#39433) (#39978) 
cherry-pick #39433 to branch-2.1
 2024-08-27 20:07:15 +08:00
meiyi
34d80f6239 [fix](group commit) Fix group commit error log when decommission (#39899) (#39955) 
pick https://github.com/apache/doris/pull/39899
 2024-08-27 18:33:24 +08:00
Mryange
4127eec9a7 [fix](function) fix error return type in mod(float32,BigInt) (#39358) (#39971) 
## Proposed changes
https://github.com/apache/doris/pull/39358
```
CREATE TABLE testdb (
    K1 BIGINT,
    K2 FLOAT
) properties("replication_num" = "1");
insert into testdb values(1,1.1);
select mod(k1,k2) from testdb;
mysql [test10]>select mod(k1,k2) from testdb;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Function mod get failed, expr is VectorizedFnCall[mod](arguments=K1, K2,return=Nullable(Float32)) and return type is Nullable(Float32).
```

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-27 18:32:52 +08:00
amory
d70fe130d7 [fix](explode-map) fix explode_map with alias (#39972) 
## Proposed changes
backport: https://github.com/apache/doris/pull/39757
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-27 18:32:10 +08:00
amory
fc9936d923 [fix](funcs) fix map struct construct funcs (#39973) 
## Proposed changes
backport: https://github.com/apache/doris/pull/39699
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-27 18:31:51 +08:00
zclllhhjj
da292b6908 [Fix-2.1](testcase) fix an unstable semi-structure case result (#39981) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-27 18:31:32 +08:00
Mingyu Chen
173aafc86f [Enhancement] add information_schema.table_properties #38745 (#38746) (#39886) 
bp #38746

---------

Co-authored-by: Vallish Pai <vallishpai@gmail.com>
 2024-08-27 17:22:19 +08:00
Mingyu Chen
bbf7701046 [fix](catalog) hide some properties in show create catalog stmt (#39946) (#39970) 
bp #39946
 2024-08-27 16:46:12 +08:00
qiye
06a2182d89 [fix](build index)Only get file size for inverted index (#39966) 
## Proposed changes

Fix `NOT FOUND` index file error message in logs.
bp #39965
 2024-08-27 15:30:51 +08:00
lihangyu
bf7675dc94 [Pick](PreparedStatement) allow prepare mysql command in Nerids even … (#39960) 
…if enableServeSidePreparedStatement is false to avoid fallback to
legacy planner

backport #39465
 2024-08-27 14:58:28 +08:00
camby
b7e0bfa1c2 [improve](function) opt aes_encrypt/decrypt function to handle const column (#37194) (#39954) 
pick #37194 to branch-2.1

---------

Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com>
 2024-08-27 14:56:38 +08:00
amory
4cf769b39f [Improve](table-function) explode json array with json args (#39491) 2024-08-27 14:53:17 +08:00
seawinde
a4c5c03acf [fix](mtmv) Add debug log decide for performance when query rewrite by materialized view (#39914) (#39952) 
## Proposed changes
In method `AbstractMaterializedViewRule#isMaterializationValid` Should
add `LOG.isDebugEnabled()` before print debug log. Because
`Plan#treeString` in debug log is performance consume.

## Proposed changes

pr: https://github.com/apache/doris/pull/39914 
commitId: 45ddb8ce
 2024-08-27 14:27:44 +08:00
Xujian Duan
dd509d0d35 [cherry-pick](branch-2.1)fix regression test fail (#39951) 
## Proposed changes

Issue Number: close #xxx
cherry-pick master #39935


<!--Describe your changes.-->
 2024-08-27 13:20:31 +08:00
yujun
959ea278cb [fix](replay) fix replay ReplicaPersistInfo no update lastFailedVersion and lastSuccVersion #39918 (#39947) 
cherry pick from #39918
 2024-08-27 11:58:13 +08:00
zzzxl
8dc3f3f347 [fix](inverted index) Fix Session Variable Compatibility (#39939) 2024-08-27 10:17:47 +08:00
Mingyu Chen
8256c6f0ba [Fix](parquet-reader) Fix definition level rle decode dead loop in parquet-reader. (#39523) (#39945) 
bp #39523

Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
 2024-08-27 08:54:43 +08:00
zclllhhjj
db0724dfe0 [Fix-2.1](function) fix function covar core for not null input (#39943) 
## Proposed changes

Issue Number: close #xxx

add testcases like:
```groovy
    qt_notnull1 "select covar_samp(non_nullable(x), non_nullable(y)) from test_covar_samp"
    qt_notnull2 "select covar_samp(x, non_nullable(y)) from test_covar_samp"
    qt_notnull3 "select covar_samp(non_nullable(x), y) from test_covar_samp"
```

before they will all coredump in 2.1
 2024-08-27 08:39:47 +08:00
zhangstar333
21bd4a4ac8 [bug](function)fix json_replace check return type error (#37014) (#39938) 
1. fix the return type dcheck error:
```
mysql [test]>select (json_replace(a, '$.fparam.nested_2', "qwe")) from json_table_2 limit 1;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INTERNAL_ERROR]Function json_replace get failed, expr is VectorizedFnCall[json_replace](arguments=a, String, String, String,return=Nullable(String)) and return type is Nullable(String).
```

2. improve the json_replace/json_insert/json_set function execute of not
convert const column, test about could faster 1s on 1000w table rows

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-27 08:03:48 +08:00
minghong
aacd61a98f [fix](nereids)filter estimation for slot=unknown #39592 (#39933) 
## Proposed changes
pick #39592
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-27 08:03:00 +08:00
Socrates
a5c8ed1cde [branch2.1][fix](cache) Catch the directory_iterator's error_code (#39922) 
## Proposed changes

Catch the directory_iterator's error_code to avoid exceptions causing
core dump
 2024-08-27 08:00:52 +08:00
shouchengShen
e4bf10ee96 [enhancement](Load)allow load data to the other partitions when some partitions are restoring (#39915) 
If broker load or stream load task execute in one table that is
restoring data, load task will failed with Exception.
Exception info :"Table [xxx] is under restore" or "Table [xxx] is in
restore process, can't load into it".

But mostly restoreJob only effects some partitions in this table, not
all of them, so that the other partitions still need to load data
successfully.
To achieve this goal, before checking olap table state, check partition
state first.

cherry pick from master branch, pr has been merged:
https://github.com/apache/doris/pull/39595

Co-authored-by: shenshoucheng <shenshoucheng@jd.com>
 2024-08-26 21:04:34 +08:00
walter
fb5002d723 [fix](binlog) Fix NPE when recover binlogs (#39909) (#39910) 
Cherry-pick #39909 

The field partition ID of DropPartitionInfo was added in PR:
apache/doris#37196, the old version doesn't contain this field so it
will be null.
 2024-08-26 20:17:26 +08:00
Gabriel
a6e485e54f [fix](followup) Fix wrong partition num (#39917) 2024-08-26 18:25:36 +08:00
Sun Chenyang
44c565840d [fix] (regression test) fix error tests (#39907) 
## Proposed changes

fix error tests: `test_dup_table_inverted_index`,
`test_single_compaction_with_variant_inverted`
 2024-08-26 17:48:52 +08:00
walter
9412ba56ae [fix](restore) Reset the db name of the materialized index stmt #39710 (#39848) 
cherry pick from #39710, #39855
 2024-08-26 16:51:50 +08:00
wangbo
357394bb3e [branch-2.1]Reset io limit default value (#39898) 
pick #39842
 2024-08-26 14:27:35 +08:00
Gabriel
5acd1279a9 [fix](2.1) Fix correctness in branch-2.1 (#39901) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-26 14:12:59 +08:00
walter
e21d2b82d6 [chore](backup) Fix the db name of the restored view (#38072) 
Cherry-pick #37412, #39855

Previously, during restore, the database name in the CREATE VIEW
statement was not modified, causing the restored view to be unviewable
with the SHOW VIEW command. This PR retains the original cluster's
database name in the BackupMeta and manually replaces it with the new
cluster's database name in the CREATE VIEW statement during restore.
 2024-08-26 14:12:47 +08:00
zhangdong
d87a220d2b [opt](privilege) Grant check name (#39597) (#39856) 
pick https://github.com/apache/doris/pull/39597
 2024-08-26 09:53:45 +08:00
Yongqiang YANG
4c1c67e03a [improvemen](overflow) Provide the user with a suggestion to avoid th… (#39631) (#39897) 
cherry-pick #39631 to branch-2.1
 2024-08-26 08:10:32 +08:00
yiguolei
d563621f6e [enhancement](thrift) add value number to thrift definition for enum type (#39880) (#39896) 
Issue Number: close #xxx
 cherry pick : https://github.com/apache/doris/pull/39880

---------

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-08-26 08:07:57 +08:00
Yongqiang YANG
f2a37d58fb [fix](stat) handle overflow of memory stat if load failed (#39621) (#39887) 
## Proposed changes

pick #39621 

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-25 18:24:08 +08:00
zclllhhjj
f7b6ed772b [Fix-2.1](testcase) Fix some testcases unstable results due to lack of orderby (#39892) 
## Proposed changes

Issue Number: close #xxx

some sql's order-by column has same value, add more order-by column to
fix it.
 2024-08-25 08:45:36 +08:00
zzzxl
44b80fb03b [fix](inverted index) Fix Session Variable Compatibility (#39884) 
https://github.com/apache/doris/pull/39889
 2024-08-25 08:42:36 +08:00
TengJianPing
8dbd73988a [fix](recvr) catch exception of transmit_block (#39882) 
BP #39881
 2024-08-25 00:25:20 +08:00
yiguolei
e0534c9bfc [bugfix](thrift) the definition number should consistent with master branch (#39879) 
## Proposed changes
introduced by pr https://github.com/apache/doris/pull/35103

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-08-25 00:22:19 +08:00