database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,622 Commits 1 Branch 0 Tags
2425730609dfa80b6d15f208aae8a5d4abfc4ef6
Commit Graph

571 Commits

Author SHA1 Message Date
HappenLee
28998300d4 [Bug](fix) fix ubsan use int32_t pointer access bool value (#38621) 
## Proposed changes

Issue Number: close #38617

<!--Describe your changes.-->
 2024-08-01 13:52:12 +08:00
Gabriel
c93f3bd24e [Improvement](bloom filter) Forbid small bloom filter (#38349) (#38392) 
Bloom filter has a expected filter ratio when data is enough. This PR
forbid too small bloom filter which has a big bias for filter ratio.

pick #38349
 2024-07-26 10:11:31 +08:00
yiguolei
4b31e52b24 [enhancement](runtimefilter) fix potential core in runtime filter sync filter size (#38058) (#38093) 
pick #38058

## Proposed changes
IRuntimeFilter maybe deconstructed before the rpc finished, so that
could not use a raw pointer in closure. Has to use the context's shared
ptr.

---------

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-07-18 23:11:26 +08:00
Pxl
e2c2702dff [Bug](runtime-filter) fix some rf error problems (#37155) 
## Proposed changes
pick from #37273
 2024-07-04 20:03:46 +08:00
Pxl
70e1c563b3 [Chore](runtime-filter) enlarge sync filter size rpc timeout limit (#37103) (#37225) 
pick from #37103
 2024-07-03 21:02:26 +08:00
Pxl
cb80ae906f [Bug](runtime-filter) disable sync filter when pipeline engine is off (#36994) 
## Proposed changes
1. disable sync filter when pipeline engine is off
2. reduce some warning log
 2024-06-28 16:59:26 +08:00
Pxl
db2721915e [Bug](runtime-filter) release dependency when rf rpc failed or meet error status (#36297) 
pick from #36126
 2024-06-14 23:44:08 +08:00
Pxl
e962a7309b [Chore](runtime-filter) adjust some check and error msg on runtime filter (#35018) (#35251) 
adjust some check and error msg on runtime filter
 2024-05-23 11:20:02 +08:00
Pxl
84f7bfffe2 [Bug](bitmap-filter) fix empty bitmap when rf do merge (#34182) 
fix empty bitmap when rf do merge
 2024-05-22 14:29:50 +08:00
HappenLee
8ca399ab92 [exec](pipeline) runtime filter wait time (#35108) 2024-05-21 12:50:05 +08:00
Pxl
0d106fe4c2 [Bug](runtime-filter) release rf count dependency when query canceled (#34367) 
* release rf count dependency when query canceled

* update

* update
 2024-05-02 09:56:17 +08:00
Pxl
7fbca522b7 [Bug](runtime-filter) fix bloom filter size error on rf merge (#34082) 
fix bloom filter size error on rf merge

W20240424 11:28:56.826277 3494287 ref_count_closure.h:80] RPC meet error status: [INVALID_ARGUMENT]PStatus: (172.21.0.15)[INVALID_ARGUMENT]bloom filter size not the same: already allocated bytes 65536, expected allocated bytes 32768
 2024-04-26 07:41:56 +08:00
zclllyybb
25358564ca [Fix](compile) Fix gcc compile on master (#33864) 
This is imported by #33511. wrongly used

ColumnStr<T> ();

which violate C++20 standard(see https://wg21.cmeerw.net/cwg/issue2237) but still supported by clang up until now(see llvm/llvm-project#58112)
 2024-04-19 23:41:37 +08:00
Pxl
175e85d616 [Bug](runtime-filter) fix coredump on no null string type rf (#33869) 
fix coredump on no null string type rf
 2024-04-19 15:03:06 +08:00
HappenLee
1300317723 [Exec](join) Support column string64 to avoid join failed in string size overflow the uint32 (#33511) (#33850) 2024-04-18 19:43:08 +08:00
Pxl
9bcb23351c [Improvement](runtime-filter) make sync rf size work when need_local_merge (#33717) 
make sync rf size work when need_local_merge
 2024-04-17 23:42:14 +08:00
Gabriel
690bf54346 [runtime filter](fix) Fix wrong results caused by IN_OR_BLOOM filter (#33701) (#33715) 2024-04-17 23:42:12 +08:00
Pxl
918f7225b4 [Bug](runtime-filter) make need_local_merge unrelated with broadcast and support merge on bitmap_filter (#33664) 2024-04-17 23:42:12 +08:00
Pxl
3081fc584d [Improvement](runtime-filter) support sync join node build side's size to init bloom runtime filter (#32180) 
support sync join node build side's size to init bloom runtime filter
 2024-04-11 09:31:50 +08:00
Pxl
6412753517 [improve](exec) reduce copy on store_string_ref (#33232) 2024-04-10 16:00:12 +08:00
Pxl
8fd6d4c41b [Chore](build) add -Wconversion and remove some unused code (#33127) 
add -Wconversion and remove some unused code
 2024-04-10 15:26:08 +08:00
Gabriel
a8232c67f9 [pipelineX](runtime filter) Fix task timeout caused by runtime filter (#33332) (#33369) 2024-04-08 16:30:32 +08:00
Pxl
05a84bd485 [Bug](runtime-filter) set need_local_merge to false when rf is broadcast (#33211) 
set need_local_merge to false when rf is broadcast
 2024-04-03 19:14:09 +08:00
Pxl
34f5521643 [Bug](min-max) store string data in MinMaxNumFunc to avoid use after free when cancel (#33152) 
* store string data in MinMaxNumFunc to avoid use after free when cancel

* update
 2024-04-02 22:35:58 +08:00
HappenLee
b6a35d68b0 [code](Refactor) Del unless filter id in runtime filter func (#32502) 
Del unless filter id in runtime filter func
 2024-03-21 14:07:49 +08:00
HappenLee
6ea8e51261 [Performance](join) speed up the colocate and bucket shuffle join by change rf size (#32421) 2024-03-21 14:07:24 +08:00
HappenLee
b66840efd7 [Fix](regression test) Fix <=> rf cause regresion test failed (#32377) 2024-03-21 14:07:23 +08:00
HappenLee
9c1888e7ec [RuntimeFilter](exec) support min max runtime filter and do refactor (#32210) 2024-03-15 18:06:20 +08:00
HappenLee
c8f3643890 [exec](runtimefilter) support null aware in runtime filter (#32152) 
null aware in runtime filter
 2024-03-15 18:05:13 +08:00
zclllyybb
847ec368be [Fix](smooth-upgrade) Fix incompatibility when upgrade from 2.0 to 2.1 (#32220) 2024-03-14 11:23:05 +08:00
Mryange
b0b7161ad0 [feature](rf) add filter info profile when rf run as expr (#31822) 2024-03-12 14:17:48 +08:00
HappenLee
2470634859 [RuntimeFilter] fix <=> runtime filter failed bug (#32003) 2024-03-12 14:13:13 +08:00
HappenLee
231768db0d [Performance](exec) Support runtime filter in <=> join (#31754) 2024-03-06 13:06:26 +08:00
HappenLee
b248d3a27e [Refactor](rf) Refactor the rf code interface to remove update filter v1 (#31643) 2024-03-02 17:12:49 +08:00
Uniqueyou
d8b9909675 [Fix](Status) Handle returned Status correctly #31434 2024-03-01 04:25:43 +08:00
HappenLee
52c45e38af [Refactor](RF) refactor the profile of rf and pipeline-x support local ignore (#31287) 
* [Refactor](RF) refactor the profile of rf and pipeline-x support local ignore

* fix local merge filter
 2024-02-23 19:05:06 +08:00
HappenLee
c56cb0ac3e [Exec](RF) Support merge remote rf local first (#31067) 2024-02-22 13:01:48 +08:00
lihangyu
278b232e76 [Bug](json reader) object should stop processing when encounter error (#31159) 
If DATA_QUALITY_ERROR encountered we should stop processing this document any more.Otherwise there will be UB in simdjson.
 2024-02-21 13:53:32 +08:00
yiguolei
a3c78dd21a [chore](refactor) refactor some rf code and delete rpc file (#31031) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-02-18 11:50:17 +08:00
HappenLee
45b4189bb6 [Refactor](opt) Opt rf and remove unless code (#30900) 
Opt rf and remove unless code
 2024-02-18 11:50:16 +08:00
yiguolei
2573150f6d [refactor](runtime filter) do not wait runtime filter rpc finished when hash node or pipeline finished (#30970) 
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-02-16 10:16:40 +08:00
HappenLee
02c37b8ead opt the rf code and remove rf unless code (#30861) 2024-02-16 10:12:24 +08:00
Pxl
0f47f7f389 [Feature](runtime filter) normalize ignore runtime filter (#30152) 
normalize ignore runtime filter
 2024-02-03 20:24:39 +08:00
HappenLee
711b156a78 [Refactor][Rf] remove unless code in RF (#30597) 2024-01-31 23:53:40 +08:00
lihangyu
e6fbccd3ed [Feature](Variant) support row store for variant type (#30052) 2024-01-31 23:53:39 +08:00
HappenLee
6231300e9e [Fix](Rf) fix in_or_bloom filter merge error in broadcast join remote target tpcds q78 (#30492) 2024-01-29 19:03:47 +08:00
HappenLee
2befa75b9c [Rf](exec) Support build exactly not cal by ndv (#30398) 2024-01-27 09:09:02 +08:00
lihangyu
9aaa6ba351 [Fix](Variant) fix variant lost null info after cast_column (#30153) 
This could result incorrect output in hirachinal cases

```
 sql """insert into ${table_name} values (-3, '{"a" : 1, "b" : 1.5, "c" : [1, 2, 3]}')"""
    sql """insert into  ${table_name} select -2, '{"a": 11245, "b" : [123, {"xx" : 1}], "c" : {"c" : 456, "d" : "null", "e" : 7.111}}'  as json_str
            union  all select -1, '{"a": 1123}' as json_str union all select *, '{"a" : 1234, "xxxx" : "kaana"}' as json_str from numbers("number" = "4096") limit 4096 ;"""

mysql> select v["c"] from var_rs where k = -3 or k = -2;
+----------------------+
| element_at(`v`, 'c') |
+----------------------+
| [1,2,3]              |
| []                   |
+----------------------+
2 rows in set (0.04 sec)
```
 2024-01-27 09:08:29 +08:00
HappenLee
92a9e8265b [Imporve](RF) Support remote broadcast join build bf exactly (#30247) 2024-01-23 14:11:54 +08:00
Gabriel
a4f29193f6 [pipelineX](fix) Fix incorrect runtime filter (#29860) 2024-01-12 13:58:19 +08:00