4ee661202e
[improve](transaction) extend abort transaction time ( #28662 )
2023-12-21 14:01:05 +08:00
d11bb11592
[enhancement](backup/restore) support alter s3 repo info about ak/sk/token ( #27027 )
...
in some cases:
s3.session_token/AWS_TOKEN will be expired after hours, and may be upload snapshot job will failed if data is big large;
as same reason, repo will be expired too when RepositoryMgr exec repo ping
so it need support alter s3 repo properties about ak/sk/token and update upload snapshot job properties to continue backup.
Signed-off-by: nextdreamblue <zxw520blue1@163.com >
2023-12-21 13:21:55 +08:00
223454d1db
[feature](Nereids) elimite inner join by foreign key ( #28486 )
2023-12-21 12:58:55 +08:00
db523dafcb
[improve](move-memtable) limit task num in load stream flush token ( #28748 )
2023-12-21 12:19:58 +08:00
425bbaa4fd
[test](Nereids): add assert debug log for TopnToMaxTest ( #28755 )
2023-12-21 12:01:15 +08:00
e9848066c9
[FIX](type) fix matchExactType for complex type ( #28233 )
...
fe matchExactType function should call type.matchTypes for its own logic, do not switch case to do special logic otherwise we may meet core in be like this.
```
F20231208 18:54:39.359673 680131 block.h:535] Check failed: _data_types[i]->is_nullable() target type: Struct(l_info:Nullable(Array(Nullable(String)))) src type: Struct(col:Nullable(Array(Nullable(UInt8))))
*** Check failure stack trace: ***
@ 0x5584e952b926 google::LogMessage::SendToLog()
@ 0x5584e9527ef0 google::LogMessage::Flush()
@ 0x5584e952c169 google::LogMessageFatal::~LogMessageFatal()
@ 0x5584cf17201e doris::vectorized::MutableBlock::merge_impl<>()
@ 0x5584ceac4b1d doris::vectorized::MutableBlock::merge<>()
@ 0x5584d4dd7de3 doris::vectorized::VUnionNode::get_next_const()
@ 0x5584d4dd9a45 doris::vectorized::VUnionNode::get_next()
@ 0x5584bce469bd std::__invoke_impl<>()
@ 0x5584bce466d0 std::__invoke<>()
@ 0x5584bce465c7 _ZNSt5_BindIFMN5doris8ExecNodeEFNS0_6StatusEPNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS1_St12_PlaceholderILi1EESC_ILi2EESC_ILi3EEEE6__callIS2_JOS4_OS7_OS8_EJLm0ELm1ELm2ELm3EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
@ 0x5584bce46358 std::_Bind<>::operator()<>()
@ 0x5584bce46208 std::__invoke_impl<>()
@ 0x5584bce46178 _ZSt10__invoke_rIN5doris6StatusERSt5_BindIFMNS0_8ExecNodeEFS1_PNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS3_St12_PlaceholderILi1EESD_ILi2EESD_ILi3EEEEJS5_S8_S9_EENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESL_E4typeEOSM_DpOSN_
@ 0x5584bce45c18 std::_Function_handler<>::_M_invoke()
@ 0x5584bce6412f std::function<>::operator()()
@ 0x5584bce56382 doris::ExecNode::get_next_after_projects()
@ 0x5584bce26218 doris::PlanFragmentExecutor::get_vectorized_internal()
@ 0x5584bce2431b doris::PlanFragmentExecutor::open_vectorized_internal()
@ 0x5584bce22a96 doris::PlanFragmentExecutor::open()
@ 0x5584bce27c9d doris::PlanFragmentExecutor::execute()
@ 0x5584bcbdb3f8 doris::FragmentMgr::_exec_actual()
@ 0x5584bcbf982f doris::FragmentMgr::exec_plan_fragment()::$_0::operator()()
@ 0x5584bcbf9715 std::__invoke_impl<>()
@ 0x5584bcbf96b5 _ZSt10__invoke_rIvRZN5doris11FragmentMgr18exec_plan_fragmentERKNS0_23TExecPlanFragmentParamsERKSt8functionIFvPNS0_12RuntimeStateEPNS0_6StatusEEEE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESH_E4typeEOSI_DpOSJ_
@ 0x5584bcbf942d std::_Function_handler<>::_M_invoke()
@ 0x5584b9dfd883 std::function<>::operator()()
@ 0x5584bd6e3929 doris::FunctionRunnable::run()
@ 0x5584bd6cf8ce doris::ThreadPool::dispatch_thread()
```
2023-12-21 11:49:05 +08:00
12089cda08
[refactor](nereids) make NormalizeAggregate rule more clear and readable ( #28607 )
2023-12-21 11:42:22 +08:00
62833006d6
[test](Nereids): remove shape plan project and distribute in eager test ( #28724 )
2023-12-21 11:14:26 +08:00
34fd376f33
[fix](publish version) fix publish fail but return ok ( #28425 )
2023-12-21 11:10:08 +08:00
9d1a0b5612
[improve](rpc) Log channel state before shutdown backend service client ( #28667 )
2023-12-21 10:40:56 +08:00
42c5dba9df
[fix](fe ut) fix PropertyConverterTest ( #28722 )
2023-12-21 10:39:54 +08:00
0e0ad3c4b1
[fix](regression) fix stream load properties case fail ( #28680 )
2023-12-21 10:39:19 +08:00
bcf2683b9d
[fix](scanner) fix concurrency bugs when scanner is stopped or finished ( #28650 )
...
`ScannerContext` will schedule scanners even after stopped, and confused with `_is_finished` and `_should_stop`.
Only Fix the concurrency bugs when scanner is stopped or finished reported in https://github.com/apache/doris/pull/28384
2023-12-21 10:37:58 +08:00
970e1c8475
[fix](group_commit) fix group commit cancel stuck ( #28749 )
2023-12-21 10:32:21 +08:00
5bf77c8f74
[Fix] (schema change) forbid adding time type column ( #28751 )
2023-12-21 10:27:50 +08:00
e4a58b7407
[Feature](materialized-view) support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg ( #28747 )
...
support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg
2023-12-21 10:27:33 +08:00
007f152e5e
[Improve](compile) add __AVX2__ macro for JsonbParser ( #28754 )
...
* [Improve](compile) add `__AVX2__` macro for JsonbParser
* throw exception instead of CHECK
2023-12-21 10:25:26 +08:00
8759bce426
[fix](stmt):fix CreateTableStmt toSql placed comment in wrong place ( #27504 )
...
Issue Number: close #27474
Co-authored-by: tongyang.han <tongyang.han@jiduauto.com >
2023-12-21 09:57:20 +08:00
a443a39e2c
[enhance](blacklist) seperate blacklist conf from heartbeat ( #28638 )
...
There is a circuit breaker lasting for 2 minutes in grpc, then if a be is down and up again, send fragments to the be fails lasting for 2 minutes.
2023-12-21 00:17:45 +08:00
18ad8562f2
[refactor](broadcastbuffer) using a queue to remove ref and unref codes ( #28698 )
...
Co-authored-by: yiguolei <yiguolei@gmail.com >Add a new class broadcastbufferholderqueue to manage holders
Using shared ptr to manage holders, not use ref and unref, it is too difficult to maintain.
2023-12-20 21:23:25 +08:00
a8dcca98ec
[FIX](explode)fix explode array decimal ( #28744 )
...
* fix explode with array<decimal> has specific precision at old planner
2023-12-20 20:19:56 +08:00
280a01b815
[pipelineX](improvement) Support global runtime filter ( #28692 )
2023-12-20 20:06:26 +08:00
504693be7f
[bug](coredump) Fix coredump in aggregation node's destruction( #28684 )
...
fix coredump in aggregation node's destruction
2023-12-20 20:02:48 +08:00
36857006cd
[Fix](json reader) fix json reader crash due to fmt::format_to ( #28737 )
...
```
4# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
5# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
6# 0x00005622F33D22B1 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
7# 0x00005622F33D2404 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
8# fmt::v7::detail::error_handler::on_error(char const*) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
9# char const* fmt::v7::detail::parse_replacement_field<char, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&>(char const*, char const*, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
10# void fmt::v7::detail::vformat_to<char>(fmt::v7::detail::buffer<char>&, fmt::v7::basic_string_view<char>, fmt::v7::basic_format_args<fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<fmt::v7::type_identity<char>::type>, fmt::v7::type_identity<char>::type> >, fmt::v7::detail::locale_ref) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
11# doris::vectorized::NewJsonReader::_append_error_msg(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:924
12# doris::vectorized::NewJsonReader::_set_column_value
```
2023-12-20 19:58:30 +08:00
ea646c6682
[fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE ( #28723 )
...
when replay addTaskResult log,will create one ConnectContext,and set Env.getCurrentEnv,then store this ctx in ConnectContext.threadLocalInfo,threadLocalInfo is static,so this ctx can not be recycling，Env of replay thread also can not be recycling
2023-12-20 19:46:58 +08:00
7b96730e87
[fix](block) fix nullptr in MutableBlock::allocated_bytes ( #28738 )
2023-12-20 19:46:13 +08:00
49eed98c1e
[fix](tvf)Fixed the avro-scanner projection pushdown failing to query on multiple BEs ( #28709 )
2023-12-20 19:39:26 +08:00
40afee0097
[fix](Nereids) delete partition failed ( #28717 )
...
1. parser's partitionSpec changed unexpectly by PR #26492
2. delete without using should support un-equals expression
2023-12-20 19:31:53 +08:00
e8d0569d8b
[refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline ( #28488 )
2023-12-20 19:26:00 +08:00
d1cc6d9a85
[test](regression-test) order by decs should only make effect on its nearest column #28728
2023-12-20 19:06:41 +08:00
9a5ec43f05
[fix](nereids) Fix data wrong using mv rewrite and ignore case when getting mv related partition table ( #28699 )
...
1. Fix data wrong using mv rewrite
2. Ignore case when getting mv related partition table
3. Enable infer expression column name without alias when create mv
2023-12-20 17:59:46 +08:00
2b2d3d0eb1
[fix](meta_scanner) fix meta_scanner process ColumnNullable ( #28711 )
2023-12-20 17:41:38 +08:00
2683b1e8c3
[test](Nereids): add test for scalar agg ( #28712 )
2023-12-20 17:35:04 +08:00
b631c51e6a
[FIX](regresstest) fix case with double value #28668
...
double value in case has 27 figures after the point, which will make not standable output
2023-12-20 17:26:28 +08:00
08066a55f1
[Enhancement](auto-partition) change the behaviour when insert overwrite an auto partition table #28683
...
If we specific target partition(s) when inserting overwrite an auto partition table,
before:
could create new partition
now:
behalf just like non-auto partition table
2023-12-20 17:19:21 +08:00
552a5c1632
[fix](mtmv) fix insert overwrite getExecTimeout error ( #28700 )
...
should use InsertTimeout but QueryTimeout
2023-12-20 17:08:38 +08:00
9d5b9cc452
[fix](statistics)Fix drop stats fail silently bug. ( #28635 )
...
Drop stats use IN predicate to filter the column stats to delete. The default length of IN predicate is 1024, drop table stats with more than 1024 columns may fail.
This pr is to split the delete sql based on the IN predicate length.
2023-12-20 15:41:25 +08:00
2a81d238f5
[test](Nereids): remove shape plan project and distribute in eager test ( #28701 )
2023-12-20 14:41:49 +08:00
0c9c32c52d
[Feature](datatype) update be ut codes and fix bugs for IPv4/v6 ( #28670 )
2023-12-20 14:38:46 +08:00
bcc32b5b26
[feature](invert index) match_regexp feature added ( #28257 )
2023-12-20 14:30:35 +08:00
32964879f7
[opt](inverted index) Add null document interface to optimize empty string indexing ( #28661 )
2023-12-20 12:11:02 +08:00
afd55123ea
[fix](partial update) report error directly if missing rowsets during doing alignment when flushing memtable due to compaction ( #28677 )
2023-12-20 11:49:11 +08:00
fe184e322a
[code](pipelineX) refine some pipelineX code ( #28570 )
2023-12-20 11:45:06 +08:00
3e85797443
[fix](Nereids) should only do bind relation in view analyzer ( #28637 )
2023-12-20 10:47:51 +08:00
0ae87e705e
[fix](planner) ctas update datev1 to datev2 should use equals ( #28641 )
2023-12-20 10:31:49 +08:00
23784787b1
[fix](planner)should bind expr using no slot to correct tuple ( #28656 )
2023-12-20 10:29:59 +08:00
c26c0c31a5
[refactor](runtimefilter) do not use QueryContext in runtime filter ( #28559 )
2023-12-20 10:28:55 +08:00
4c0080e237
[feat](Nereids) support outer join and aggregate bitmap rewrite by mv ( #28596 )
...
- Support left outer join rewrite by materialized view
- Support bitmap_union roll up to imp count(distinct)
- Support partition materialized view rewrite
2023-12-20 10:23:30 +08:00
17268346d3
[Revert](partial update) Revert "Fix missing rowsets during doing alignment when flushing memtable due to compaction ( #28062 )" ( #28674 )
...
This reverts commit 485d7db5161efb011b43a10237df1ca83d5e5c3d.
2023-12-20 10:13:43 +08:00
e59c707022
[fix](stream_load)fix bug for stream ( #27752 )
...
1. forbid thed stream_load without content-length or chunked Transfer Encoding
2. forbid thed stream_load both with content-length and chunked Transfer Encoding
Co-authored-by: xingying01 <xingying01@corp.netease.com >
2023-12-20 09:47:52 +08:00
