d969047b50
[Refactor](join) refactor of hash join ( #27557 )
...
Improve the performance under the tpch data set by reconstructing the join related code and the use of hash table
Co-authored-by: HappenLee <happenlee@hotmail.com >
Co-authored-by: BiteTheDDDDt <pxl290@qq.com >
2023-11-28 19:46:00 +08:00
f565f60bc3
[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default ( #27587 )
2023-11-28 13:02:30 +08:00
b457856bd2
[chore](be) remove bthread scanner related codes ( #27417 )
2023-11-23 15:18:49 +08:00
3e1a5b6e29
[profile](rf) make runtime filter profile readable ( #27323 )
2023-11-22 21:13:35 +08:00
3ad865fef9
[refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits ( #26191 )
2023-11-15 21:34:49 +08:00
30d1e6036c
[feature](runtime filter) New session variable runtime_filter_wait_infinitely ( #26888 )
...
New session variable: runtime_filter_wait_infinitely. If set runtime_filter_wait_infinitely = true, consumer of rf will wait on receiving until query is timeout.
2023-11-14 21:05:59 +08:00
504ec324bb
Revert "[refactor](scan) delete bloom_filter_predicate ( #26499 )" ( #26851 )
...
This reverts commit 2bb3ef198144954583aea106591959ee09932cba.
2023-11-13 16:27:23 +08:00
a5565f68b2
[Refactor](opentelemetry) Remove opentelemetry ( #26605 )
2023-11-09 18:05:34 +08:00
44b51bf0b9
[Feature](Variant) support variant load ( #26572 )
2023-11-08 00:37:57 -06:00
2bb3ef1981
[refactor](scan) delete bloom_filter_predicate ( #26499 )
2023-11-07 19:37:31 +08:00
9d83948987
[feature](pipelineX) Use dependency instead of block queue in the runtime filter ( #26078 )
2023-10-31 22:44:18 +08:00
c1d64a7128
[Feature](datatype) Add IPv4/v6 data type for doris ( #24965 )
2023-10-26 17:33:28 +08:00
693982fd1a
[feature](decimal) support decimal256 ( #25386 )
2023-10-25 15:47:51 +08:00
215c1625b2
[chore](fragmentid) rename fragment id to fragment instance id in thrift definition file ( #25787 )
...
* rename fragment id to fragment instance id in thrift definition file
* f
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2023-10-24 11:26:52 +08:00
2e2d5bcba2
[Improvements](status) catch some error status ( #25677 )
...
catch some error status
2023-10-23 10:19:08 +08:00
305c884478
[fix](rf) fix minmax runtime filter ( #25720 )
...
Template arguments for decimalv3 type is wrong when creating MinMaxNumFunc, which result in wrong decimal type like vectorized::Decimal<doris::vectorized::Decimal128I> :
class MinmaxFunctionTraits {
public:
using BasePtr = MinMaxFuncBase*;
template <PrimitiveType type, size_t N>
static BasePtr get_function() {
return new MinMaxNumFunc<std::conditional_t<
type == TYPE_DECIMAL32 || type == TYPE_DECIMAL64 || type == TYPE_DECIMAL128I,
vectorized::Decimal<typename PrimitiveTypeTraits<type>::CppType>,
typename PrimitiveTypeTraits<type>::CppType>>();
}
};
(gdb) bt
#0 doris::vectorized::Decimal<__int128>::operator<=>(doris::vectorized::Decimal<__int128> const&) const (x=..., this=<optimized out>) at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381
#1 doris::vectorized::Decimal<doris::vectorized::Decimal128I>::operator<=>(doris::vectorized::Decimal<doris::vectorized::Decimal128I> const&) const (x=..., this=<optimized out>)
at /mnt/disk2/user/doris-1.1/be/src/vec/core/types.h:381
#2 doris::MinMaxNumFunc<doris::vectorized::Decimal<doris::vectorized::Decimal128I> >::insert (this=0x7ef19de903c0, data=0x7c612928d860) at /mnt/disk2/user/doris-1.1/be/src/exprs/minmax_predicate.h:61
#3 0x0000558465c2844a in doris::RuntimePredicateWrapper::insert (this=0x7f47a2e3d720, value=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:492
#4 doris::RuntimePredicateWrapper::insert_batch (this=0x7f47a2e3d720, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:504
#5 0x0000558465c1f6c3 in doris::IRuntimeFilter::insert_batch (this=<optimized out>, column=..., rows=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter.cpp:1060
#6 0x000055846883dafd in doris::VRuntimeFilterSlots::insert (this=0x7d7e767f4e70, datas=...) at /mnt/disk2/user/doris-1.1/be/src/exprs/runtime_filter_slots.h:188
#7 0x0000558468840535 in doris::vectorized::ProcessRuntimeFilterBuild::operator()<doris::vectorized::MethodOneNumber<doris::vectorized::UInt128, PartitionedHashMapTable<HashMapTable<doris::vectorized::UInt128, HashMapCell<doris::vectorized::UInt128, doris::vectorized::RowRefList, HashCRC32<doris::vectorized::UInt128>, HashTableNoState>, HashCRC32<doris::vectorized::UInt128>, PartitionedHashTableGrower<8ul>, Allocator<true, true, false> > > >, doris::vectorized::HashJoinNode> (
this=this@entry=0x7e7c6a5bee40, state=0x7ef19df00400, hash_table_ctx=..., parent=0x7ef19dee3c00) at /mnt/disk2/user/doris-1.1/be/src/vec/exec/join/vhash_join_node.h:92
2023-10-23 10:16:28 +08:00
642c149e6a
remove datetime_value and move vecdatetime_value to doris namespace ( #25695 )
...
remove datetime_value and move vecdatetime_value to doris namespace
2023-10-20 22:08:17 +08:00
7385602b19
[bug](rf) fix only min/max rf return error when has remote target ( #25588 )
2023-10-19 19:26:29 +08:00
3d1206d325
[date](fix) modify push-down predicate for datev1 type ( #25571 )
...
For comparison predicate, two arguments must be cast to datetime and push down to storage if either one is date type. This PR disables predicate push-down for this case.
2023-10-19 14:18:27 +08:00
013eafc1d7
[Enhancement](filter) support only min/max runtime filter in BE ( #25290 )
...
this PR #25193 have achieve about FE.
eg: select count() from lineorder join supplier on lo_partkey < s_suppkey;
will have a max filter after build hash table , so could use it to filter probe table data.
2023-10-12 16:59:52 +08:00
58d96ecdbf
[Improve](status) avoid print too may stack log for DATA_QUALITY_ERROR code ( #25292 )
2023-10-12 09:58:51 +08:00
642e5cdb69
[Fix](Status) Make Status [[nodiscard]] and handle returned Status correctly ( #23395 )
2023-09-29 22:38:52 +08:00
35c5d71549
[Improvement](join) some improvement of hash join ( #23972 )
...
some improvement of hash join
2023-09-14 17:55:35 +08:00
d3f1388717
[Feature](partitions) Support auto-partition ( #24153 )
...
Co-authored-by: zhangstar333 <2561612514@qq.com >
2023-09-12 15:23:15 +08:00
0b24bd6a42
[Bug](pipelineX) init runtime filter profile at first ( #24106 )
2023-09-08 20:01:02 +08:00
fdb7a44f57
Revert "[Feature](partitions) Support auto partition" ( #24024 )
...
* Revert "[Feature](partitions) Support auto partition (#23236 )"
This reverts commit 6c544dd2011d731b8c9c51384c77bcf19c017981.
* Update config.h
2023-09-07 17:08:26 +08:00
6c544dd201
[Feature](partitions) Support auto partition ( #23236 )
...
Co-authored-by: zhangstar333 <2561612514@qq.com >
2023-09-06 16:26:45 +08:00
e1090d6a63
[Fix](column predicate) seperate CHAR primitive type for column predicate ( #23581 )
2023-09-01 09:41:53 +08:00
6fe2418cfc
[fix](filter) fix error id in bloomfilter ( #23564 )
...
1. "set" may overwrite the original ID.
2.A bloom filter may not necessarily be an IN_OR_BLOOM_FILTER.
before may be
RuntimeFilterInfo id -1: [type = BF, input = 25, filtered = 0]
now
RuntimeFilterInfo id 0: [type = BF, input = 25, filtered = 0]
2023-08-31 21:12:09 +08:00
d22290e548
[pipelineX](join) support hash join ( #23689 )
2023-08-31 13:01:26 +08:00
8ed4045df9
[Chore](primitive-type) remove VecPrimitiveTypeTraits ( #22842 )
2023-08-23 08:37:40 +08:00
72e264dd59
[fix](executor)fix error when FixedContainer with null ( #22850 )
2023-08-11 17:20:50 +08:00
b64f62647b
[runtime filter](profile) add merge time on non-pipeline engine ( #22363 )
2023-07-31 12:52:42 +08:00
f7e0479605
[Chore](refactor) remove some unused code ( #22152 )
...
remove some unused code
2023-07-28 17:30:46 +08:00
560731f392
[Bug](runtime-filter) fix probe expr prepared twice on minmax runtime filter ( #22229 )
...
fix probe expr prepared twice on minmax runtime filter
2023-07-26 19:44:35 +08:00
28b714c371
[feature](executor) using fe version to set instance_num ( #22047 )
2023-07-25 14:37:42 +08:00
7891c99e9f
[fix](pipeline) fix wrong state of runtime filter of pipeline ( #22179 )
2023-07-25 11:29:09 +08:00
99bf901607
[fix](in) throw exception for unsupported data type of in expr ( #22050 )
2023-07-24 14:13:31 +08:00
2c16fe0da9
[bugfix](runtimefilter) runtime filter is shared between multi instances with same node id, should not cache exprs ( #22114 )
...
runtime filter is shared among multi instances.
in the past, we cached pushdown expr(runtime filter generated)
every scannode[runtime filter consumer] will try to call prepare expr
but the expr may generated with different fn_context_id
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2023-07-23 13:04:33 +08:00
40299d280d
[Fix](json reader) fix rapidjson array->PushBack may take ownership… ( #21988 )
...
With bellow json path
`["$.data","$.data.datatimestamp"]`
After `array_obj->PushBack` the `data` field owner will be taken from array_obj, and lead to null values for json path `$.data.datatimestamp`
Rapidjson doc:
```
//! Append a GenericValue at the end of the array.
\note The ownership of \c value will be transferred to this array on success.
*/
GenericValue& PushBack(GenericValue& value, Allocator& allocator);
```
2023-07-21 17:02:01 +08:00
7a6ae12ebb
[imporve](bloomfilter) refactor runtime_filter_mgr with bloomfilter and fix bug in change_to_bloom_filter ( #21783 )
2023-07-14 17:47:32 +08:00
1a08c81adc
[Profile](runtimefilter) fix merge time of runtime filter ( #21654 )
2023-07-10 16:16:05 +08:00
f02bec8ad1
[Chore](runtime filter) change runtime filter dcheck to error status or exception ( #21475 )
...
change runtime filter dcheck to error status or exception
2023-07-05 14:03:55 +08:00
5f760a8939
[fix](runtime_filter) remove incorrect DCHECK ( #21050 )
2023-06-21 11:27:53 +08:00
ab32299ba4
[feature](nereids) Support multi target rf #20714
...
Support multi target runtime filter, mainly for set operation, such as union/intersect/except.
2023-06-16 20:26:00 +08:00
17a395f5e3
[Bug](runtime-filter) fix runtime filter not register on vdata_gen_scan_node ( #20787 )
...
fix runtime filter not register on vdata_gen_scan_node
2023-06-15 14:06:14 +08:00
a0d4f11667
[Bug](function) catch error state in function cast to avoid core dump ( #20751 )
...
catch error state in function cast to avoid core dump
2023-06-14 17:34:34 +08:00
9244cb6553
[Chore](runtime-filter) do not make query fail when rf publish failed ( #20742 )
...
do not make query fail when rf publish failed
2023-06-13 18:23:46 +08:00
e010fa8d4f
[Chore](runtime filter) remove runtime filter ready_for_publish/publish_finally ( #20593 )
2023-06-13 11:20:49 +08:00
a15a0b9193
[Chore](build) use file(GLOB_RECURSE xxx CONFIGURE_DEPENDS) to replace set cpp ( #20461 )
...
use file(GLOB_RECURSE xxx CONFIGURE_DEPENDS) to replace set cpp
2023-06-08 19:36:21 +08:00
