Commit Graph

3308 Commits

Author SHA1 Message Date
ebe031c019 [fix](inverted index) Fix match_regexp to correctly handle empty string patterns (#40659)
https://github.com/apache/doris/pull/39503
2024-09-11 18:10:33 +08:00
8708fae420 [fix](ES Catalog)Support parse single value for array column (#40614) (#40660)
bp #40614
2024-09-11 17:26:48 +08:00
d554f600bc [branch-2.1](partition) Support use Auto and Dynamic partition at the same time (#39580) (#40649)
pick https://github.com/apache/doris/pull/39580
2024-09-11 15:35:20 +08:00
3246baa451 [branch-2.1](function) Refine crypto functions signature to fix wrong result(#40285) (#40648)
pick https://github.com/apache/doris/pull/40285
2024-09-11 15:32:19 +08:00
86647df45b [fix] (inverted index) fix error result in complex compound expr (#40630)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-09-11 15:27:40 +08:00
db8fb66dba [fix](mtmv)fix nested mtmv not refresh (#40433) (#40560)
pick: https://github.com/apache/doris/pull/40433
2024-09-10 11:51:41 +08:00
8eda15ae16 [opt](routine load) support routine load perceived schema change (#39412) (#40508)
pick #39412

At present, if the table structure changes, the routine load cannot
perceive it. As a long-running load, it should be able to perceive the
changes in the table structure.
2024-09-10 11:05:58 +08:00
f69063ea87 [Fix](Variant) use uinque id to access column reader (#39841) (#40269)
#39841
#40295
2024-09-09 18:01:12 +08:00
8f37eccbf2 [Cherry-pick](branch-2.1) Pick "[Featrue](default value) Support bitmap_empty default value (#40364)" (#40487)
## Proposed changes

Pick #40364 

<!--Describe your changes.-->
2024-09-09 16:57:38 +08:00
44a7efff4f [branch-2.1] Picks "[Opt](delete) Skip newly inserted rows check in non-strict mode partial update if the row's delete sign is marked #40322" (#40383)
picks https://github.com/apache/doris/pull/40322
2024-09-09 16:32:24 +08:00
314f6ae823 [fix](ES Catalog)Fix int parse error when querying by doc_values (#40385) (#40521)
bp #40385
2024-09-09 14:29:21 +08:00
c32d9a129a [test](mtmv) SSB mv rewrite test use little data set for test performance (#40188) (#40437)
## Proposed changes

commitId: 0baa9366
pr: https://github.com/apache/doris/pull/40188
2024-09-09 11:23:47 +08:00
a67f20f073 [opt](mtmv) Support to contain select constant clause when create async materialized view (#40244) (#40435)
## Proposed changes
commitId: 518a0fc0
pr: https://github.com/apache/doris/pull/40244
2024-09-09 11:23:15 +08:00
ecb75c2e7d [fix](mtmv) Mtmv support set both immediate and starttime (#39573) (#40418)
pick: https://github.com/apache/doris/pull/39573
2024-09-09 11:13:51 +08:00
2023eab11e [Fix](ShortCircuit) consider delete sign flag when hits row (#40300) (#40408)
https://github.com/apache/doris/pull/40300
2024-09-09 10:04:05 +08:00
962c382077 [fix](jdbc catalog) Fix type recognition error when using query tvf to query doris (#40481)
pick  (#40122)

Using string to match Doris type will not work with query tvf, so use
field matching instead
2024-09-06 19:30:32 +08:00
8104b992d1 [fix](ES Catalog)Do not extract doc_values of field with ignore_above setting (#40314) (#40464)
bp #40314
2024-09-06 16:25:30 +08:00
cb0613e249 [fix] (inverted index) fix error result in compound query (#40425)
## Proposed changes

`select count() from table where a + b > 0 or b > 0`


![image](https://github.com/user-attachments/assets/df56bb36-660d-4b4f-8e38-4eebcaa09e51)



- When _execute_predicates_except_leafnode_of_andnode is executed, an
Expr tree is traversed from bottom to top. When it reaches the leaf node
b, the information of this column b is placed into new_predicate_info.

- However, this step is skipped directly at an ADD node, which leads to
the GT node at the upper level generating a sign equivalent to b > 0,
the same as the sign on the right side b > 0.

- This causes the compound OR calculation to assume that both GT
conditions below have been evaluated, thus prematurely computing this
EXPR, when in fact, the ADD node has not been evaluated.

- If the SQL is written as SELECT COUNT(*) FROM table WHERE b + a > 0 OR
b > 0, the calculation would be correct because the sign generated by
this > node would be equivalent to a > 0, which is different from b > 0
on the right side.
2024-09-06 10:27:59 +08:00
0928c9c6ed [fix](unary function) Fix wrong result of asin, acos and sqrt when processing invalid input #40267 (#40358)
cherry pick from #40267
2024-09-05 19:51:01 +08:00
c6771da240 [fix](nereids) fix bug for A>n, where A.max is infinity #39936 (#40368)
## Proposed changes
pick #39936 
Issue Number: close #xxx

<!--Describe your changes.-->
2024-09-05 08:33:04 +08:00
41271ecba0 [fix](ES Catalog)Do not push down limit to ES when predicates can not be processed by ES. (#40111) (#40265)
bp #40111
2024-09-03 11:17:24 +08:00
70eeda9f50 [cherry-pick](branch-21) fix encryption_function return wrong result at const column (#40201) (#40247)
## Proposed changes
cherry-pick from master #40201
the string object have error of order, should be src, key,iv,mode
2024-09-02 16:09:15 +08:00
c8d3202595 [regression-test](cases) optimize some cases (#40240)
#40174
2024-09-02 14:50:48 +08:00
e4ee0e8379 [fix](topn) Fix wrong rows returned by TOPN sorter (#40243)
## Proposed changes

pick #40241

<!--Describe your changes.-->
2024-09-02 14:34:53 +08:00
b104e933cd [Bug](expr) execute expr should use local states instead of operators (#40189) (#40219)
## Proposed changes

cherry-pick from master #40189

<!--Describe your changes.-->
2024-09-01 00:41:10 +08:00
d792a62409 [fix](regression) fix regression failure of window_funnel caused by duplicated table name (#40216)
BP #40217
2024-08-31 20:58:20 +08:00
163193b1d4 [branch-2.1](function) fix random_bytes return same data for multi rows (#39891) (#40137)
pick https://github.com/apache/doris/pull/39891

Issue Number: close #xxx

before:
```sql
mysql [optest]>SELECT random_bytes(10) a FROM numbers("number" = "10");
+------------------------+
| a                      |
+------------------------+
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
+------------------------+
```

now:
```sql
mysql [optest]>SELECT random_bytes(10) a FROM numbers("number" = "10");
+------------------------+
| a                      |
+------------------------+
| 0xd82cf60825b29ef2a0fd |
| 0x6f8c808415bdbaa6d257 |
| 0x7c26b5214297a151c25c |
| 0x43f02c77293063900437 |
| 0x5e5727569dec5e24f96b |
| 0x434f20bf74d7759640b7 |
| 0x087ed96b739750c733a6 |
| 0xdf05f6d7ede4972eb846 |
| 0xcefab471912264b5c54f |
| 0x1bddc019409d1926aa10 |
+------------------------+
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-30 10:43:42 +08:00
70daa1f85d [opt](inverted index) Controls whether the in_list can execute fast_execute. (#40141)
https://github.com/apache/doris/pull/40022
2024-08-30 10:32:43 +08:00
ca07a00c93 Revert "[branch-2.1](hive) support hive write text table (#38549) (#4… (#40157)
…0063)"

This reverts commit c6df7c21a3c09ae1664deabacb88dfcea9d94b68.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-08-30 10:25:38 +08:00
0b9817ea4b [cherry-pick](branch-2.1) Pick "[Enhancement](txn) Block new insert into if schema change happens during transaction (#39483)" (#40115)
## Proposed changes

Pick #39483 

<!--Describe your changes.-->
2024-08-30 10:01:10 +08:00
a7156ee775 [fix](parquet)Fix the be core issue when reading parquet unsigned types. (#39926) (#40123)
bp #39926
2024-08-29 21:52:52 +08:00
c6df7c21a3 [branch-2.1](hive) support hive write text table (#38549) (#40063)
1. Support write hive text table
2. Add SessionVariable `hive_text_compression` to write compressed hive
text table
3. Supported compression type: gzip, bzip2, snappy, lz4, zstd

pick from https://github.com/apache/doris/pull/38549
2024-08-29 16:50:40 +08:00
eb6a040a03 [branch-2.1] Picks "[Fix](partial update) Fix wrongly update autoinc column in partial update #39996" (#40073)
## Proposed changes

pick https://github.com/apache/doris/pull/39996
2024-08-29 16:41:51 +08:00
c5d9e178be [fix](mtmv) Fix result wrong when query rewrite by mv if query contains null_unsafe equals expression (#39629) (#40041)
## Proposed changes

commitId: 5d4ad028
pr: https://github.com/apache/doris/pull/39629
2024-08-29 00:31:36 +08:00
c3c6a4fc4c [pick](Variant) variant fallthrough with inverted index (#40070)
backport #40069
2024-08-29 00:30:37 +08:00
bb709ad917 [branch-2.1] Picks "[Fix](merge-on-write) Fix duplicate key problem after adding sequence column for merge-on-write table #39958" (#40010)
## Proposed changes

picks https://github.com/apache/doris/pull/39958
2024-08-28 23:00:31 +08:00
b052f46944 [fix](json-quote) fix json quote func for not find the func (#39931) (#40061)
1.  fix function not found before this pr:
```
drop table if exists t003;
create table t003 (a bigint, b json not null) properties ("replication_num"="1");
insert into t003 values (1, '{"a":1,"b":2}');
select a, map_agg("k1", json_quote(b)) from t003 group by a;

[17:47]>  select a, map_agg("k1", json_quote(b)) from t003 group by a;
(1105, 'errCode = 2, detailMessage = (172.20.48.119)[INTERNAL_ERROR]Function json_quote get failed, expr is VectorizedFnCall[json_quote](arguments=(CAST b(JSONB) TO String),return=Nullable(String)) and return type is Nullable(String).')
``` 
after pr , we can carry on it 
2. fix a core if we use json_quote with table column
```
[WARNING!] /sys/kernel/mm/transparent_hugepage/enabled: [always] madvise never, Doris not recommend turning on THP, which may cause the BE process to use more memory and cannot be freed in time. Turn off THP: `echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled`
start BE in local mode
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
*** Query id: b7b94dead55e4090-8c8e94f65ec9efd3 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1724744881 (unix time) try "date -d @1724744881" if you are using GNU date ***
*** Current BE git commitID: bd5844ea0d ***
*** SIGABRT unknown detail explain (@0x461003611e1) received by PID 3543521 (TID 3548858 OR 0x7f8a53c1a700) from PID 3543521; stack trace: ***
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/wangqiannan/amory/doris/be/src/common/signal_handler.h:421
 1# 0x00007F9429595B50 in /lib64/libc.so.6
 2# gsignal in /lib64/libc.so.6
 3# __GI_abort in /lib64/libc.so.6
 4# _nl_load_domain.cold.0 in /lib64/libc.so.6
 5# 0x00007F942958E426 in /lib64/libc.so.6
 6# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::Prefix(rapidjson::Type) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488
 7# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::String(char const*, unsigned int, bool) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:206
 8# bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u> >(rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>&) const at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/document.h:1974
 9# doris::vectorized::FunctionJsonQuoteImpl::execute(std::vector<doris::vectorized::ColumnStr<unsigned int> const*, std::allocator<doris::vectorized::ColumnStr<unsigned int> const*> > const&, doris::vectorized::ColumnStr<unsigned int>&, unsigned long) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:826
10# doris::vectorized::FunctionJson<doris::vectorized::FunctionJsonQuoteImpl>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:985
11# doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:463
12# doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:120
13# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:245
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-28 21:31:08 +08:00
4127eec9a7 [fix](function) fix error return type in mod(float32,BigInt) (#39358) (#39971)
## Proposed changes
https://github.com/apache/doris/pull/39358
```
CREATE TABLE testdb (
    K1 BIGINT,
    K2 FLOAT
) properties("replication_num" = "1");
insert into testdb values(1,1.1);
select mod(k1,k2) from testdb;
mysql [test10]>select mod(k1,k2) from testdb;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Function mod get failed, expr is VectorizedFnCall[mod](arguments=K1, K2,return=Nullable(Float32)) and return type is Nullable(Float32).
```

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-27 18:32:52 +08:00
d70fe130d7 [fix](explode-map) fix explode_map with alias (#39972)
## Proposed changes
backport: https://github.com/apache/doris/pull/39757
Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-27 18:32:10 +08:00
fc9936d923 [fix](funcs) fix map struct construct funcs (#39973)
## Proposed changes
backport: https://github.com/apache/doris/pull/39699
Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-27 18:31:51 +08:00
da292b6908 [Fix-2.1](testcase) fix an unstable semi-structure case result (#39981)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-27 18:31:32 +08:00
173aafc86f [Enhancement] add information_schema.table_properties #38745 (#38746) (#39886)
bp #38746

---------

Co-authored-by: Vallish Pai <vallishpai@gmail.com>
2024-08-27 17:22:19 +08:00
4cf769b39f [Improve](table-function) explode json array with json args (#39491) 2024-08-27 14:53:17 +08:00
db0724dfe0 [Fix-2.1](function) fix function covar core for not null input (#39943)
## Proposed changes

Issue Number: close #xxx

add testcases like:
```groovy
    qt_notnull1 "select covar_samp(non_nullable(x), non_nullable(y)) from test_covar_samp"
    qt_notnull2 "select covar_samp(x, non_nullable(y)) from test_covar_samp"
    qt_notnull3 "select covar_samp(non_nullable(x), y) from test_covar_samp"
```

before they will all coredump in 2.1
2024-08-27 08:39:47 +08:00
21bd4a4ac8 [bug](function)fix json_replace check return type error (#37014) (#39938)
1. fix the return type dcheck error:
```
mysql [test]>select (json_replace(a, '$.fparam.nested_2', "qwe")) from json_table_2 limit 1;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INTERNAL_ERROR]Function json_replace get failed, expr is VectorizedFnCall[json_replace](arguments=a, String, String, String,return=Nullable(String)) and return type is Nullable(String).
```

2. improve the json_replace/json_insert/json_set function execute of not
convert const column, test about could faster 1s on 1000w table rows

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-27 08:03:48 +08:00
e21d2b82d6 [chore](backup) Fix the db name of the restored view (#38072)
Cherry-pick #37412, #39855

Previously, during restore, the database name in the CREATE VIEW
statement was not modified, causing the restored view to be unviewable
with the SHOW VIEW command. This PR retains the original cluster's
database name in the BackupMeta and manually replaces it with the new
cluster's database name in the CREATE VIEW statement during restore.
2024-08-26 14:12:47 +08:00
f7b6ed772b [Fix-2.1](testcase) Fix some testcases unstable results due to lack of orderby (#39892)
## Proposed changes

Issue Number: close #xxx

some sql's order-by column has same value, add more order-by column to
fix it.
2024-08-25 08:45:36 +08:00
9997911ec9 [feat](Nereids) support nereids hint position detaction (#39113) (#39417)
cherry-pick: #39113
When use hint in wrong position or use unsupport hint, use channel(2) to
filter it out

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-24 23:59:54 +08:00
263746b04b [fix](paimon) fix crash when enable cache with paimon deletion vector(#39877) (#39875)
bp #39877
2024-08-24 17:58:20 +08:00
460605ae3c [branch-2.1] pick some prs (#39860)
## Proposed changes

Issue Number: close #xxx

https://github.com/apache/doris/pull/38385 optimize parsing datetime
https://github.com/apache/doris/pull/38978 make stream load failure
message more clear and disable some error's stacktrace by default
https://github.com/apache/doris/pull/39255 fix random function coredump
https://github.com/apache/doris/pull/39324 fix function corr
inconsistency with doc
https://github.com/apache/doris/pull/39449 check auto partitoin nullity
when creating partition
https://github.com/apache/doris/pull/39695 make
DynamicPartitionScheduler immediately know interval's change
https://github.com/apache/doris/pull/39754 Add some partition expr check
on creating table
2024-08-24 17:26:42 +08:00