Commit Graph

296 Commits

Author SHA1 Message Date
5deeb42f6d branch-2.1: [opt] Optimization for short circuit of CompoundPred #45422 (#46241)
cherry pick from #45422
2025-01-02 10:11:47 +08:00
8e730faec5 [Exec](expr) Opt the compound pred performace (#45414) (#46232)
cherry-pick #45414

before:
```
 mysqlslap -hd3 -uroot -P9130  --create-schema=test_db2 -c 10 -i 500 -q "SELECT count(k) FROM sbtest1_dup WHERE k BETWEEN 4850578 AND 8454295 OR k BETWEEN 8776291 AND 29749077;"
Benchmark
	Average number of seconds to run all queries: 0.041 seconds
	Minimum number of seconds to run all queries: 0.037 seconds
	Maximum number of seconds to run all queries: 0.115 seconds
	Number of clients running queries: 10
	Average number of queries per client: 1
```

after:
```
mysqlslap -hd3 -uroot -P9030  --create-schema=test_db -c 10 -i 500 -q "SELECT count(k) FROM sbtest1 WHERE k BETWEEN 4850578 AND 8454295 OR k BETWEEN 8776291 AND 29749077;"
Benchmark
	Average number of seconds to run all queries: 0.029 seconds
	Minimum number of seconds to run all queries: 0.027 seconds
	Maximum number of seconds to run all queries: 0.034 seconds
	Number of clients running queries: 10
	Average number of queries per client: 1
```

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-01-01 16:19:42 +08:00
Pxl
43c646363e [Bug](runtime-filter) support ip rf and use exception to replace dche… (#41531)
…ck when PrimitiveType to PColumnType (#39985)

use exception to replace dcheck when PrimitiveType to PColumnType
```cpp
*** SIGABRT unknown detail explain (@0x11d3f) received by PID 73023 (TID 74292 OR 0x7fd758225640) from PID 73023; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
 1# 0x00007FDDBE6B9520 in /lib/x86_64-linux-gnu/libc.so.6
 2# pthread_kill at ./nptl/pthread_kill.c:89
 3# raise at ../sysdeps/posix/raise.c:27
 4# abort at ./stdlib/abort.c:81
 5# 0x000056123F81A94D in /root/output/be/lib/doris_be
 6# 0x000056123F80CF8A in /root/output/be/lib/doris_be
 7# google::LogMessage::SendToLog() in /root/output/be/lib/doris_be
 8# google::LogMessage::Flush() in /root/output/be/lib/doris_be
 9# google::LogMessageFatal::~LogMessageFatal() in /root/output/be/lib/doris_be
10# doris::to_proto(doris::PrimitiveType) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:114
11# doris::IRuntimeFilter::push_to_remote(doris::TNetworkAddress const*) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:1143
12# doris::IRuntimeFilter::publish(bool)::$_0::operator()(doris::IRuntimeFilter*) const at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:959
13# doris::IRuntimeFilter::publish(bool)::$_2::operator()() const at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:983
14# doris::IRuntimeFilter::publish(bool) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:997
```

## Proposed changes
pick from #39985
2024-12-30 20:56:11 +08:00
f0031d9954 [cherry-pick](branch-21)support posexplode table function (#43221) (#45783)
cherry-pick from master #43221
2024-12-24 21:42:30 +08:00
367ecc3292 [fix](expr)Remove the _can_fast_execute flag from VExpr. (#45542) (#45662) 2024-12-22 21:26:32 +08:00
79662fcc94 [branch-2.1](functions) clean some ip functions code and make IS_IP_ADDRESS_IN_RANGE DEPENDS_ON_ARGUMENT (#45358)
pick https://github.com/apache/doris/pull/35239


add special logic to deal smooth upgrade

The origin PR is https://github.com/apache/doris/pull/35239. for
branch-3.0 it was merged in 3.0.0 but forgot to register old version.
now in branch-3.0 we fix it in
https://github.com/apache/doris/pull/45428 which must be merged in
3.0.4. and do same thing in this PR which must be merged in 2.1.8.
then:
```
FROM    TO    result
217-    218+    
217-    303-    💥
218+    303-    
218+    304+    
303-    304+    
```
this is our best result.
2024-12-17 11:51:07 +08:00
fb407f2e94 [opt](lambda) let lambda expression support refer outer slot (#45186) 2024-12-11 18:55:49 +08:00
625a1ea6ad branch-2.1: [Optimize](Expr) Opt getting value of VLitreal (#43249)
Cherry-picked from #43204

Co-authored-by: zclllhhjj <zhaochangle@selectdb.com>
2024-11-10 01:03:56 +08:00
64d0c55e1d [fix](decimal256) support decimal256 for many functions (#42136) (#42353)
BP #42136
2024-10-24 10:52:28 +08:00
ff52e73a07 [Fix](inverted index) fix match null for inverted index #41746 (#41787)
cherry pick from #41746
2024-10-14 14:45:36 +08:00
31b506c8cc [Enhancement](inverted index) return OK instead of not supported in expr evaluate_inverted_index #41567 (#41578)
cherry pick from #41567
2024-10-09 15:14:38 +08:00
b3bb4f4402 [Enhancement](inverted index) improve expr evaluate_inverted_index performace and remove useless code #40600 (#41442)
cherry pick from #40600
2024-09-29 12:00:58 +08:00
727f0374be [Refactor](inverted index) refactor inverted index compound predicates evaluate logic #38908 (#41385)
cherry pick from #38908
2024-09-29 09:19:17 +08:00
6961c95eca [chore](error msg) More convenient error msg when function not found. #40296 (#40570)
cherry pick from #40296
2024-09-10 12:03:12 +08:00
70daa1f85d [opt](inverted index) Controls whether the in_list can execute fast_execute. (#40141)
https://github.com/apache/doris/pull/40022
2024-08-30 10:32:43 +08:00
15b14ef49b [fix](inverted index) fix error handling in fast_execute (#40086)
https://github.com/apache/doris/pull/40024
2024-08-29 14:45:25 +08:00
4cf769b39f [Improve](table-function) explode json array with json args (#39491) 2024-08-27 14:53:17 +08:00
273a62584c [opt](inverted index) unified optimization judgment to prevent omissions (#39473)
https://github.com/apache/doris/pull/38027
2024-08-17 16:57:19 +08:00
b0da8430bc [opt](inverted index) Optimize the usage of the multi_match function (#39472)
## Proposed changes

https://github.com/apache/doris/pull/39193

<!--Describe your changes.-->
2024-08-17 16:53:52 +08:00
60eeec3754 [fix] (inverted index) Fix match function without inverted index (#38989) (#39220)
## Proposed changes

pick from #38989
2024-08-13 10:55:54 +08:00
5e1e725cee [feature](inverted index) Add multi_match function #37722 #38931 #39149 (#38877) 2024-08-10 15:20:08 +08:00
ff6fa33021 [opt](inverted index) mow supports index optimization #(#38180)
## Proposed changes

https://github.com/apache/doris/pull/37428
https://github.com/apache/doris/pull/37429

<!--Describe your changes.-->
2024-08-06 11:18:13 +08:00
66ebf709ba [Fix](inverted index) fix fast execute for not_in expr #37745 (#38594)
cherry pick from #37745
2024-07-31 19:58:12 +08:00
c6cd1e2e3d [opt](inverted index) opt value extraction from column to string (#38179)
## Proposed changes

https://github.com/apache/doris/pull/37395
2024-07-24 15:38:41 +08:00
21c6b854f7 [fix](explode-json-object)fix explode json object (#37956)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-17 01:03:07 +08:00
3cb1d4e842 [feature](json)support explode_json_object func #36887 (#37378) 2024-07-16 10:59:11 +08:00
1d49d386aa [cherry-pick](branch-21) remove the useless code in column vector (#34432) (#37827)
cherry-pick from master https://github.com/apache/doris/pull/34432

Co-authored-by: HappenLee <happenlee@hotmail.com>
2024-07-15 22:10:58 +08:00
967173d7d0 [cherry-pick-2.1](table-function) pick some table functions exec performance (#34090) (#37778)
## Proposed changes

pick from master:
https://github.com/apache/doris/pull/33904
https://github.com/apache/doris/pull/34090

Co-authored-by: HappenLee <happenlee@hotmail.com>
2024-07-15 17:15:56 +08:00
04e62d9c42 [fix](invert index) ensure that the pred result sign of the inlist is in order #36085 (#36191) 2024-06-12 23:04:31 +08:00
f67bd4b03b [Fix](inverted index) fix fast execute condition for vexpr (#35673) 2024-06-01 11:24:54 +08:00
680be6d19f [fix](ub) fix uninitialized accesses in BE (#35370)
ubsan hints:
```c++
/root/doris/be/src/olap/hll.h:93:29: runtime error: load of value 3078029312, which is not a valid value for type 'HllDataType'
/root/doris/be/src/olap/hll.h:94:23: runtime error: load of value 3078029312, which is not a valid value for type 'HllDataType'
/root/doris/be/src/runtime/descriptors.h:439:38: runtime error: load of value 118, which is not a valid value for type 'bool'
/root/doris/be/src/vec/exec/vjdbc_connector.cpp:61:50: runtime error: load of value 35, which is not a valid value for type 'bool' 
```
2024-05-29 20:31:07 +08:00
bef931de9d [fix](inverted index) add in list to fast execute logic after hit index (#35344)
resolve the issue where the case fails when
enable_common_expr_pushdown is set to false
2024-05-29 20:30:43 +08:00
df96f76f78 [featrue](pipelineX) check output type in some node (#33716) 2024-04-24 17:13:49 +08:00
7f61626c8d [fix](arrow_flight_sql) Fix ArrowSchema column alias (#33490)
run: select TABLE_SCHEMA as a, sum(TABLE_ROWS) as b  from tables group by TABLE_SCHEMA limit 2;
old output:

          TABLE_SCHEMA                        Nullable(Int64)_1
0  regression_test_mv_p0_sum_count           9
1  regression_test_query_p0_sql_functions_string_functions       70414
now output:

          a                        b
0  regression_test_mv_p0_sum_count            9
1  regression_test_query_p0_sql_functions_string_functions       70414
2024-04-22 11:28:22 +08:00
25358564ca [Fix](compile) Fix gcc compile on master (#33864)
This is imported by #33511. wrongly used

ColumnStr<T> ();

which violate C++20 standard(see https://wg21.cmeerw.net/cwg/issue2237) but still supported by clang up until now(see llvm/llvm-project#58112)
2024-04-19 23:41:37 +08:00
0ac7849a9d [exec](table_fun) opt bitmap/split vexplode table func performance (#33876) 2024-04-19 15:22:14 +08:00
1300317723 [Exec](join) Support column string64 to avoid join failed in string size overflow the uint32 (#33511) (#33850) 2024-04-18 19:43:08 +08:00
ea19224d14 [exec](table_fun) opt numbers table func performance (#33804) 2024-04-18 19:04:03 +08:00
272269f9c1 [Fix](inverted index) fix fast execute problem when need read data opt enabled (#33526) 2024-04-17 23:41:59 +08:00
f7d52b5b1c [feature](expr) add type check when expr prepare (#33330) 2024-04-11 09:31:50 +08:00
Pxl
8fd6d4c41b [Chore](build) add -Wconversion and remove some unused code (#33127)
add -Wconversion and remove some unused code
2024-04-10 15:26:08 +08:00
ff0da8108b [fix](RF) fix 'Invalid value' error of RF of decimal type (#32749) 2024-03-25 22:34:19 +08:00
35e580ec7a [fix](RF) fix 'Invalid value' error of RF of datetimev2 type for max value (#32649) 2024-03-22 22:29:50 +08:00
0db402e154 [expr](fix) Not to throw exception when close failed (#32287) 2024-03-21 14:07:49 +08:00
b92a764665 [feature](function) Support for aggregate function foreach combiner for some error function (#31913)
Support for aggregate function foreach combiner for some error function
2024-03-21 14:07:49 +08:00
a5f3611b88 [Fix](Regression) DCHECK failed in runtime filter wrapper (#32446) 2024-03-21 14:07:23 +08:00
9c1888e7ec [RuntimeFilter](exec) support min max runtime filter and do refactor (#32210) 2024-03-15 18:06:20 +08:00
c8f3643890 [exec](runtimefilter) support null aware in runtime filter (#32152)
null aware in runtime filter
2024-03-15 18:05:13 +08:00
ccd21a6ea4 [Improve](InPredict) enhance in predict with array type (#31828) 2024-03-12 14:19:14 +08:00
b0b7161ad0 [feature](rf) add filter info profile when rf run as expr (#31822) 2024-03-12 14:17:48 +08:00