d42fd68d6b
[opt](invert index) Empty strings are not written to the index in the case of TOKENIZED ( #28822 )
2023-12-25 10:23:07 +08:00
b7ae7a07c7
[fix](join) incorrect result of left semi/anti join with empty build side ( #28898 )
2023-12-25 09:07:38 +08:00
1d8822b2b7
[fix](paimon)fix like predicate ( #28803 )
...
fix like predict
2023-12-23 22:25:55 +08:00
96d4778f2e
[fix](parquet) the end offset of column chunk may be wrong in parquet metadata ( #28891 )
2023-12-23 22:21:04 +08:00
0a1d9f4cbc
[feature](mtmv)add more test case1 ( #28910 )
2023-12-23 14:39:44 +08:00
623257d02b
[feature](mtmv)MTMV pause and resume ( #28887 )
...
- PAUSE MATERIALIZED VIEW JOB ON mv1
- RESUME MATERIALIZED VIEW JOB ON mv1
- fix when drop db,not drop job
- add lock for one materialized view can only run one task at a time
2023-12-23 14:30:54 +08:00
e51f75e424
[FIX](map)fix map with rowstore table ( #28877 )
2023-12-23 12:11:06 +08:00
93184bc656
[refactor](test)Refactor workload group/schedule policy test #28888
...
[refactor](test)Refactor workload group/schedule policy test
#28888
2023-12-22 22:18:07 +08:00
b2b209e938
[feature](Nereids) Pull up join from union all ( #28682 )
2023-12-22 19:35:20 +08:00
3ed82bcee2
[Feature](inverted index) add lowercase option for inverted index analyzer ( #28704 )
2023-12-22 18:22:44 +08:00
5420575184
check stats and log memo for ds46 ( #28396 )
2023-12-22 14:50:11 +08:00
9b67c86219
[optimize](count) optimize pk exact query without reading data ( #28494 )
2023-12-22 14:18:15 +08:00
d1e1619e89
[feature](mtmv)mtmv partition refresh case ( #28787 )
2023-12-22 14:03:31 +08:00
7710c85904
[fix](stream load)add test case and doc for arrow type of stream load ( #28098 )
...
add test case and doc for arrow type of stream load
2023-12-22 13:18:44 +08:00
f38e11ec4e
[fix](paimon)fix type convert for paimon ( #28774 )
...
fix type convert for paimon
2023-12-22 13:18:25 +08:00
012e66729a
[improvement](executor) Add tvf and regression test for Workload Scheduler ( #28733 )
...
1 Add select workload schedule policy tvf
2 Add reg test
2023-12-22 12:09:51 +08:00
c72ad9b673
[fix](regression) fix regression error of test_compress_type ( #28826 )
2023-12-22 12:08:23 +08:00
13ccfa06a7
[Feature](Variant) Implement variant new sub column access method ( #28484 )
...
* [Feature](Variant) Implement variant new sub column access method
The query SELECT v["a"]["b"] from simple_var WHERE cast(v["a"]["b"] as int) = 1 encompasses three primary testing scenarios:
```
1. A basic test involving the variant data type.
2. A scenario dealing with GitHub event data in the context of a variant.
3. A case related to the TPC-H benchmark using a variant.
```
2023-12-22 11:59:37 +08:00
f6b6180462
[Fix](Variant) fix variant predicate rewrite OrToIn with wrong plan ( #28695 )
...
using the name without paths info will lead to wrong In plan, e.g.
```
where cast(v:a as text) = 'hello' or cast(v:b as text) = 'world'
```
will be rewrite to:
```
where cast(v as text) in ('hello', 'world')
``
This is wrong, because they are different slots
2023-12-22 11:51:36 +08:00
0af6bd6390
[fix](group-commit) check if wal need recovery is abnormal ( #28769 )
2023-12-22 11:06:11 +08:00
25dcf40f36
[test](partial update) add complex type regression cases for partial update ( #28758 )
...
NOTE: There's some issue for MAP type to work with row store, so in this PR we don't have cases for MAP type
Will add the support for MAP type in future.
2023-12-22 10:55:24 +08:00
172f68480b
[Enhancement](load) Limit the number of incorrect data drops and add documents ( #27727 )
...
In the load process, if there are problems with the original data, we will store the error data in an error_log file on the disk for subsequent debugging. However, if there are many error data, it will occupy a lot of disk space. Now we want to limit the number of error data that is saved to the disk.
Be familiar with the usage of doris' import function and internal implementation process
Add a new be configuration item load_error_log_limit_bytes = default value 200MB
Use the newly added threshold to limit the amount of data that RuntimeState::append_error_msg_to_file writes to disk
Write regression cases for testing and verification
Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com >
2023-12-22 10:43:18 +08:00
0b9b1be1f1
[fix](function) Fix from_second functions overflow and wrong result ( #28685 )
2023-12-22 10:22:49 +08:00
e51e94fdec
[fix](Nereids) join order is not right after sql parsing ( #28721 )
...
for sql
```
t1, t2 join t3
```
we should generate plan like:
```
t1 join (t2 join t3)
```
but we generate:
```
(t1 join t2) join t3
```
to follow legancy planner.
2023-12-21 20:31:40 +08:00
ba4033ad2e
[opt](Nereids) improve Shape check cases ( #28124 )
...
* tmplate update
* log tpcds stats when check shape
2023-12-21 20:19:54 +08:00
4f1aebb8e8
(topN)runtime_predicate is only triggered when the column name is obtained ( #28419 )
...
Issue Number: close #27485
2023-12-21 18:08:23 +08:00
5d8c465644
[regression](p2) fix test cases result ( #28768 )
...
regression-test/data/external_table_p2/hive/test_hive_hudi.out
regression-test/data/external_table_p2/hive/test_hive_to_array.out
regression-test/suites/external_table_p2/tvf/test_local_tvf_compression.groovy
regression-test/suites/external_table_p2/tvf/test_path_partition_keys.groovy
regression-test/data/external_table_p2/hive/test_hive_text_complex_type.out
2023-12-21 14:38:30 +08:00
e9848066c9
[FIX](type) fix matchExactType for complex type ( #28233 )
...
fe matchExactType function should call type.matchTypes for its own logic, do not switch case to do special logic otherwise we may meet core in be like this.
```
F20231208 18:54:39.359673 680131 block.h:535] Check failed: _data_types[i]->is_nullable() target type: Struct(l_info:Nullable(Array(Nullable(String)))) src type: Struct(col:Nullable(Array(Nullable(UInt8))))
*** Check failure stack trace: ***
@ 0x5584e952b926 google::LogMessage::SendToLog()
@ 0x5584e9527ef0 google::LogMessage::Flush()
@ 0x5584e952c169 google::LogMessageFatal::~LogMessageFatal()
@ 0x5584cf17201e doris::vectorized::MutableBlock::merge_impl<>()
@ 0x5584ceac4b1d doris::vectorized::MutableBlock::merge<>()
@ 0x5584d4dd7de3 doris::vectorized::VUnionNode::get_next_const()
@ 0x5584d4dd9a45 doris::vectorized::VUnionNode::get_next()
@ 0x5584bce469bd std::__invoke_impl<>()
@ 0x5584bce466d0 std::__invoke<>()
@ 0x5584bce465c7 _ZNSt5_BindIFMN5doris8ExecNodeEFNS0_6StatusEPNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS1_St12_PlaceholderILi1EESC_ILi2EESC_ILi3EEEE6__callIS2_JOS4_OS7_OS8_EJLm0ELm1ELm2ELm3EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
@ 0x5584bce46358 std::_Bind<>::operator()<>()
@ 0x5584bce46208 std::__invoke_impl<>()
@ 0x5584bce46178 _ZSt10__invoke_rIN5doris6StatusERSt5_BindIFMNS0_8ExecNodeEFS1_PNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS3_St12_PlaceholderILi1EESD_ILi2EESD_ILi3EEEEJS5_S8_S9_EENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESL_E4typeEOSM_DpOSN_
@ 0x5584bce45c18 std::_Function_handler<>::_M_invoke()
@ 0x5584bce6412f std::function<>::operator()()
@ 0x5584bce56382 doris::ExecNode::get_next_after_projects()
@ 0x5584bce26218 doris::PlanFragmentExecutor::get_vectorized_internal()
@ 0x5584bce2431b doris::PlanFragmentExecutor::open_vectorized_internal()
@ 0x5584bce22a96 doris::PlanFragmentExecutor::open()
@ 0x5584bce27c9d doris::PlanFragmentExecutor::execute()
@ 0x5584bcbdb3f8 doris::FragmentMgr::_exec_actual()
@ 0x5584bcbf982f doris::FragmentMgr::exec_plan_fragment()::$_0::operator()()
@ 0x5584bcbf9715 std::__invoke_impl<>()
@ 0x5584bcbf96b5 _ZSt10__invoke_rIvRZN5doris11FragmentMgr18exec_plan_fragmentERKNS0_23TExecPlanFragmentParamsERKSt8functionIFvPNS0_12RuntimeStateEPNS0_6StatusEEEE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESH_E4typeEOSI_DpOSJ_
@ 0x5584bcbf942d std::_Function_handler<>::_M_invoke()
@ 0x5584b9dfd883 std::function<>::operator()()
@ 0x5584bd6e3929 doris::FunctionRunnable::run()
@ 0x5584bd6cf8ce doris::ThreadPool::dispatch_thread()
```
2023-12-21 11:49:05 +08:00
12089cda08
[refactor](nereids) make NormalizeAggregate rule more clear and readable ( #28607 )
2023-12-21 11:42:22 +08:00
62833006d6
[test](Nereids): remove shape plan project and distribute in eager test ( #28724 )
2023-12-21 11:14:26 +08:00
34fd376f33
[fix](publish version) fix publish fail but return ok ( #28425 )
2023-12-21 11:10:08 +08:00
e4a58b7407
[Feature](materialized-view) support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg ( #28747 )
...
support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg
2023-12-21 10:27:33 +08:00
a8dcca98ec
[FIX](explode)fix explode array decimal ( #28744 )
...
* fix explode with array<decimal> has specific precision at old planner
2023-12-20 20:19:56 +08:00
d1cc6d9a85
[test](regression-test) order by decs should only make effect on its nearest column #28728
2023-12-20 19:06:41 +08:00
9a5ec43f05
[fix](nereids) Fix data wrong using mv rewrite and ignore case when getting mv related partition table ( #28699 )
...
1. Fix data wrong using mv rewrite
2. Ignore case when getting mv related partition table
3. Enable infer expression column name without alias when create mv
2023-12-20 17:59:46 +08:00
b631c51e6a
[FIX](regresstest) fix case with double value #28668
...
double value in case has 27 figures after the point, which will make not standable output
2023-12-20 17:26:28 +08:00
08066a55f1
[Enhancement](auto-partition) change the behaviour when insert overwrite an auto partition table #28683
...
If we specific target partition(s) when inserting overwrite an auto partition table,
before:
could create new partition
now:
behalf just like non-auto partition table
2023-12-20 17:19:21 +08:00
2a81d238f5
[test](Nereids): remove shape plan project and distribute in eager test ( #28701 )
2023-12-20 14:41:49 +08:00
bcc32b5b26
[feature](invert index) match_regexp feature added ( #28257 )
2023-12-20 14:30:35 +08:00
3e85797443
[fix](Nereids) should only do bind relation in view analyzer ( #28637 )
2023-12-20 10:47:51 +08:00
4c0080e237
[feat](Nereids) support outer join and aggregate bitmap rewrite by mv ( #28596 )
...
- Support left outer join rewrite by materialized view
- Support bitmap_union roll up to imp count(distinct)
- Support partition materialized view rewrite
2023-12-20 10:23:30 +08:00
17268346d3
[Revert](partial update) Revert "Fix missing rowsets during doing alignment when flushing memtable due to compaction ( #28062 )" ( #28674 )
...
This reverts commit 485d7db5161efb011b43a10237df1ca83d5e5c3d.
2023-12-20 10:13:43 +08:00
7da86c37ec
[fix](hive) add support for quoteChar and seperatorChar for hive ( #28613 )
...
add support for quoteChar and seperatorChar .
2023-12-19 19:35:03 +08:00
e362bf674f
[regression-test](memtable) test memtable flush is high priority for vtable writerV1 ( #28502 )
2023-12-19 12:33:12 +08:00
1922638004
[Improvement](regression) change compound predicate regression case name to make it more clear ( #28612 )
2023-12-19 12:17:39 +08:00
ddba98159e
[feature](Nereids) support datev1 and datetimev1 ( #28581 )
2023-12-19 11:31:28 +08:00
88f0bee722
[Performance](point query)Opimize partition prune for point query ( #28150 )
...
* [Performance](point query)Opimize partition prune for point query
2023-12-19 11:00:13 +08:00
6503aaf7db
[feature](planner) allow HLL and QUANTILE_STATE types on duplicate and unique table ( #28546 )
2023-12-19 09:54:24 +08:00
9cf9d568c7
[feature](Nereids): add is null in predicate and put or expansion rule in rewriter ( #28348 )
...
add is null in predicate
put or expansion rule in rewrite job
2023-12-18 20:00:32 +08:00
9ebacb1faa
[fix](expr) fix performance problem caused by too many virtual function call ( #28508 )
2023-12-18 12:01:55 +08:00
