3ad865fef9
[refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits ( #26191 )
2023-11-15 21:34:49 +08:00
30d1e6036c
[feature](runtime filter) New session variable runtime_filter_wait_infinitely ( #26888 )
...
New session variable: runtime_filter_wait_infinitely. If set runtime_filter_wait_infinitely = true, consumer of rf will wait on receiving until query is timeout.
2023-11-14 21:05:59 +08:00
504ec324bb
Revert "[refactor](scan) delete bloom_filter_predicate ( #26499 )" ( #26851 )
...
This reverts commit 2bb3ef198144954583aea106591959ee09932cba.
2023-11-13 16:27:23 +08:00
a5565f68b2
[Refactor](opentelemetry) Remove opentelemetry ( #26605 )
2023-11-09 18:05:34 +08:00
44b51bf0b9
[Feature](Variant) support variant load ( #26572 )
2023-11-08 00:37:57 -06:00
2bb3ef1981
[refactor](scan) delete bloom_filter_predicate ( #26499 )
2023-11-07 19:37:31 +08:00
9d83948987
[feature](pipelineX) Use dependency instead of block queue in the runtime filter ( #26078 )
2023-10-31 22:44:18 +08:00
c1d64a7128
[Feature](datatype) Add IPv4/v6 data type for doris ( #24965 )
2023-10-26 17:33:28 +08:00
693982fd1a
[feature](decimal) support decimal256 ( #25386 )
2023-10-25 15:47:51 +08:00
215c1625b2
[chore](fragmentid) rename fragment id to fragment instance id in thrift definition file ( #25787 )
...
* rename fragment id to fragment instance id in thrift definition file
* f
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2023-10-24 11:26:52 +08:00
2e2d5bcba2
[Improvements](status) catch some error status ( #25677 )
...
catch some error status
2023-10-23 10:19:08 +08:00
305c884478
[fix](rf) fix minmax runtime filter ( #25720 )
...
Template arguments for decimalv3 type is wrong when creating MinMaxNumFunc, which result in wrong decimal type like vectorized::Decimal<doris::vectorized::Decimal128I> :
class MinmaxFunctionTraits {
public:
using BasePtr = MinMaxFuncBase*;
template <PrimitiveType type, size_t N>
static BasePtr get_function() {
return new MinMaxNumFunc<std::conditional_t<
type == TYPE_DECIMAL32 || type == TYPE_DECIMAL64 || type == TYPE_DECIMAL128I,
vectorized::Decimal<typename PrimitiveTypeTraits<type>::CppType>,
typename PrimitiveTypeTraits<type>::CppType>>();
}
};
(gdb) bt
#0 doris::vectorized::Decimal<__int128>::operator<=>(doris::vectorized::Decimal<__int128> const&) const (x=..., this=<optimized out>) at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381
#1 doris::vectorized::Decimal<doris::vectorized::Decimal128I>::operator<=>(doris::vectorized::Decimal<doris::vectorized::Decimal128I> const&) const (x=..., this=<optimized out>)
at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381
#2 doris::MinMaxNumFunc<doris::vectorized::Decimal<doris::vectorized::Decimal128I> >::insert (this=0x7ef19de903c0, data=0x7c612928d860) at /mnt/disk2/user/doris-1.1/be/src/exprs/minmax_predicate.h:61
#3 0x0000558465c2844a in doris::RuntimePredicateWrapper::insert (this=0x7f47a2e3d720, value=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:492
#4 doris::RuntimePredicateWrapper::insert_batch (this=0x7f47a2e3d720, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:504
#5 0x0000558465c1f6c3 in doris::IRuntimeFilter::insert_batch (this=<optimized out>, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:1060
#6 0x000055846883dafd in doris::VRuntimeFilterSlots::insert (this=0x7d7e767f4e70, datas=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter_slots.h:188
#7 0x0000558468840535 in doris::vectorized::ProcessRuntimeFilterBuild::operator()<doris::vectorized::MethodOneNumber<doris::vectorized::UInt128, PartitionedHashMapTable<HashMapTable<doris::vectorized::UInt128, HashMapCell<doris::vectorized::UInt128, doris::vectorized::RowRefList, HashCRC32<doris::vectorized::UInt128>, HashTableNoState>, HashCRC32<doris::vectorized::UInt128>, PartitionedHashTableGrower<8ul>, Allocator<true, true, false> > > >, doris::vectorized::HashJoinNode> (
this=this@entry=0x7e7c6a5bee40, state=0x7ef19df00400, hash_table_ctx=..., parent=0x7ef19dee3c00) at /mnt/disk2/user/doris-1.1/be/src/vec/exec/join/vhash_join_node.h:92
2023-10-23 10:16:28 +08:00
642c149e6a
remove datetime_value and move vecdatetime_value to doris namespace ( #25695 )
...
remove datetime_value and move vecdatetime_value to doris namespace
2023-10-20 22:08:17 +08:00
7385602b19
[bug](rf) fix only min/max rf return error when has remote target ( #25588 )
2023-10-19 19:26:29 +08:00
3d1206d325
[date](fix) modify push-down predicate for datev1 type ( #25571 )
...
For comparison predicate, two arguments must be cast to datetime and push down to storage if either one is date type. This PR disables predicate push-down for this case.
2023-10-19 14:18:27 +08:00
013eafc1d7
[Enhancement](filter) support only min/max runtime filter in BE ( #25290 )
...
this PR #25193 have achieve about FE.
eg: select count() from lineorder join supplier on lo_partkey < s_suppkey;
will have a max filter after build hash table , so could use it to filter probe table data.
2023-10-12 16:59:52 +08:00
58d96ecdbf
[Improve](status) avoid print too may stack log for DATA_QUALITY_ERROR code ( #25292 )
2023-10-12 09:58:51 +08:00
642e5cdb69
[Fix](Status) Make Status [[nodiscard]] and handle returned Status correctly ( #23395 )
2023-09-29 22:38:52 +08:00
35c5d71549
[Improvement](join) some improvement of hash join ( #23972 )
...
some improvement of hash join
2023-09-14 17:55:35 +08:00
d3f1388717
[Feature](partitions) Support auto-partition ( #24153 )
...
Co-authored-by: zhangstar333 <2561612514@qq.com >
2023-09-12 15:23:15 +08:00
0b24bd6a42
[Bug](pipelineX) init runtime filter profile at first ( #24106 )
2023-09-08 20:01:02 +08:00
fdb7a44f57
Revert "[Feature](partitions) Support auto partition" ( #24024 )
...
* Revert "[Feature](partitions) Support auto partition (#23236 )"
This reverts commit 6c544dd2011d731b8c9c51384c77bcf19c017981.
* Update config.h
2023-09-07 17:08:26 +08:00
6c544dd201
[Feature](partitions) Support auto partition ( #23236 )
...
Co-authored-by: zhangstar333 <2561612514@qq.com >
2023-09-06 16:26:45 +08:00
e1090d6a63
[Fix](column predicate) seperate CHAR primitive type for column predicate ( #23581 )
2023-09-01 09:41:53 +08:00
6fe2418cfc
[fix](filter) fix error id in bloomfilter ( #23564 )
...
1. "set" may overwrite the original ID.
2.A bloom filter may not necessarily be an IN_OR_BLOOM_FILTER.
before may be
RuntimeFilterInfo id -1: [type = BF, input = 25, filtered = 0]
now
RuntimeFilterInfo id 0: [type = BF, input = 25, filtered = 0]
2023-08-31 21:12:09 +08:00
d22290e548
[pipelineX](join) support hash join ( #23689 )
2023-08-31 13:01:26 +08:00
8ed4045df9
[Chore](primitive-type) remove VecPrimitiveTypeTraits ( #22842 )
2023-08-23 08:37:40 +08:00
72e264dd59
[fix](executor)fix error when FixedContainer with null ( #22850 )
2023-08-11 17:20:50 +08:00
b64f62647b
[runtime filter](profile) add merge time on non-pipeline engine ( #22363 )
2023-07-31 12:52:42 +08:00
f7e0479605
[Chore](refactor) remove some unused code ( #22152 )
...
remove some unused code
2023-07-28 17:30:46 +08:00
560731f392
[Bug](runtime-filter) fix probe expr prepared twice on minmax runtime filter ( #22229 )
...
fix probe expr prepared twice on minmax runtime filter
2023-07-26 19:44:35 +08:00
28b714c371
[feature](executor) using fe version to set instance_num ( #22047 )
2023-07-25 14:37:42 +08:00
7891c99e9f
[fix](pipeline) fix wrong state of runtime filter of pipeline ( #22179 )
2023-07-25 11:29:09 +08:00
99bf901607
[fix](in) throw exception for unsupported data type of in expr ( #22050 )
2023-07-24 14:13:31 +08:00
2c16fe0da9
[bugfix](runtimefilter) runtime filter is shared between multi instances with same node id, should not cache exprs ( #22114 )
...
runtime filter is shared among multi instances.
in the past, we cached pushdown expr(runtime filter generated)
every scannode[runtime filter consumer] will try to call prepare expr
but the expr may generated with different fn_context_id
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2023-07-23 13:04:33 +08:00
40299d280d
[Fix](json reader) fix rapidjson array->PushBack may take ownership… ( #21988 )
...
With bellow json path
`["$.data","$.data.datatimestamp"]`
After `array_obj->PushBack` the `data` field owner will be taken from array_obj, and lead to null values for json path `$.data.datatimestamp`
Rapidjson doc:
```
//! Append a GenericValue at the end of the array.
\note The ownership of \c value will be transferred to this array on success.
*/
GenericValue& PushBack(GenericValue& value, Allocator& allocator);
```
2023-07-21 17:02:01 +08:00
7a6ae12ebb
[imporve](bloomfilter) refactor runtime_filter_mgr with bloomfilter and fix bug in change_to_bloom_filter ( #21783 )
2023-07-14 17:47:32 +08:00
1a08c81adc
[Profile](runtimefilter) fix merge time of runtime filter ( #21654 )
2023-07-10 16:16:05 +08:00
f02bec8ad1
[Chore](runtime filter) change runtime filter dcheck to error status or exception ( #21475 )
...
change runtime filter dcheck to error status or exception
2023-07-05 14:03:55 +08:00
5f760a8939
[fix](runtime_filter) remove incorrect DCHECK ( #21050 )
2023-06-21 11:27:53 +08:00
ab32299ba4
[feature](nereids) Support multi target rf #20714
...
Support multi target runtime filter, mainly for set operation, such as union/intersect/except.
2023-06-16 20:26:00 +08:00
17a395f5e3
[Bug](runtime-filter) fix runtime filter not register on vdata_gen_scan_node ( #20787 )
...
fix runtime filter not register on vdata_gen_scan_node
2023-06-15 14:06:14 +08:00
a0d4f11667
[Bug](function) catch error state in function cast to avoid core dump ( #20751 )
...
catch error state in function cast to avoid core dump
2023-06-14 17:34:34 +08:00
9244cb6553
[Chore](runtime-filter) do not make query fail when rf publish failed ( #20742 )
...
do not make query fail when rf publish failed
2023-06-13 18:23:46 +08:00
e010fa8d4f
[Chore](runtime filter) remove runtime filter ready_for_publish/publish_finally ( #20593 )
2023-06-13 11:20:49 +08:00
a15a0b9193
[Chore](build) use file(GLOB_RECURSE xxx CONFIGURE_DEPENDS) to replace set cpp ( #20461 )
...
use file(GLOB_RECURSE xxx CONFIGURE_DEPENDS) to replace set cpp
2023-06-08 19:36:21 +08:00
c7888f4bfa
[feature](profile)Add the filtering info of the in filter in profile #20321
...
image Currently, it is difficult to obtain the id of in filters,so, the some in filters's id is -1.
2023-06-06 10:24:59 +08:00
c7b8c83a7f
[Improvement](runtimefilter) Build bloom filter according to the exact build size for IN_OR_BLOOM_FILTER ( #20166 )
...
* [Improvement](runtimefilter) Build bloom filter according to the exact build size for IN_OR_BLOOM_FILTER
2023-05-30 12:55:30 +08:00
9f8de89659
[refactor](exec) replace the single pointer with an array of 'conjuncts' in ExecNode ( #19758 )
...
Refactoring the filtering conditions in the current ExecNode from an expression tree to an array can simplify the process of adding runtime filters. It eliminates the need for complex merge operations and removes the requirement for the frontend to combine expressions into a single entity.
By representing the filtering conditions as an array, each condition can be treated individually, making it easier to add runtime filters without the need for complex merging logic. The array can store the individual conditions, and the runtime filter logic can iterate through the array to apply the filters as needed.
This refactoring simplifies the codebase, improves readability, and reduces the complexity associated with handling filtering conditions and adding runtime filters. It separates the conditions into discrete entities, enabling more straightforward manipulation and management within the execution node.
2023-05-29 11:47:31 +08:00
23ad72e734
[Bug](runtime filter) Fix min/max filter for decimalv3 ( #20005 )
2023-05-26 21:35:21 +08:00
