6962de841d
[fix](test)fix test_create_table test case for nereids ( #28693 )
2023-12-22 10:55:35 +08:00
25dcf40f36
[test](partial update) add complex type regression cases for partial update ( #28758 )
...
NOTE: There's some issue for MAP type to work with row store, so in this PR we don't have cases for MAP type
Will add the support for MAP type in future.
2023-12-22 10:55:24 +08:00
172f68480b
[Enhancement](load) Limit the number of incorrect data drops and add documents ( #27727 )
...
In the load process, if there are problems with the original data, we will store the error data in an error_log file on the disk for subsequent debugging. However, if there are many error data, it will occupy a lot of disk space. Now we want to limit the number of error data that is saved to the disk.
Be familiar with the usage of doris' import function and internal implementation process
Add a new be configuration item load_error_log_limit_bytes = default value 200MB
Use the newly added threshold to limit the amount of data that RuntimeState::append_error_msg_to_file writes to disk
Write regression cases for testing and verification
Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com >
2023-12-22 10:43:18 +08:00
0b9b1be1f1
[fix](function) Fix from_second functions overflow and wrong result ( #28685 )
2023-12-22 10:22:49 +08:00
49eaf0cc32
[fix](partial update) only report error when in strict mode partial update when finding missing rowsets during flushing memtable ( #28764 )
...
related pr: #28062 , #28674 , #28677
fix #28677
2023-12-22 09:50:10 +08:00
c1457f97a4
[improvement](external catalog)Change log level to debug for getRowCountFromExternalSource. ( #28801 )
2023-12-21 22:09:50 +08:00
65e3da255f
[enhancement](audit-log) add status code and error msg to audit log for proxy stmt ( #27607 )
2023-12-21 21:16:36 +08:00
5153137b83
[fix](metrics) fix bvar memtable_input_block_allocated_size ( #28725 )
2023-12-21 21:16:14 +08:00
e51e94fdec
[fix](Nereids) join order is not right after sql parsing ( #28721 )
...
for sql
```
t1, t2 join t3
```
we should generate plan like:
```
t1 join (t2 join t3)
```
but we generate:
```
(t1 join t2) join t3
```
to follow legancy planner.
2023-12-21 20:31:40 +08:00
e74ff95087
[fix](compaction) compaction should catch exception when vertical block reader read next block ( #28625 )
2023-12-21 20:30:37 +08:00
0070909d30
[fix](group commit)Fix the issue of duplicate addition of wal path when encouter exception ( #28691 )
2023-12-21 20:27:33 +08:00
ba4033ad2e
[opt](Nereids) improve Shape check cases ( #28124 )
...
* tmplate update
* log tpcds stats when check shape
2023-12-21 20:19:54 +08:00
ee73833d6e
[improve](load) reduce lock scope in MemTableWriter active consumption ( #28790 )
2023-12-21 20:18:35 +08:00
cd65796874
[opt](inverted index) ignore_above only affects untokenized strings ( #28819 )
2023-12-21 20:06:56 +08:00
619d500333
[bug](pipelineX) Fix pipelineX bug on multiple BE ( #28792 )
2023-12-21 19:15:52 +08:00
a16680cf91
[fix](planner)fix bug of bound conjunct to wrong tuple ( #28811 )
...
this fix bug introduced by #28656
2023-12-21 19:15:36 +08:00
4f1aebb8e8
(topN)runtime_predicate is only triggered when the column name is obtained ( #28419 )
...
Issue Number: close #27485
2023-12-21 18:08:23 +08:00
5c469a8b6c
[pipelineX](fix) Fix TPCH Q2 ( #28783 )
2023-12-21 17:11:01 +08:00
f3bf26c269
[Chore](decimal) set check_overflow_for_decimal to true when alter table( #28777 )
...
set check_overflow_for_decimal to true when alter table
2023-12-21 16:27:06 +08:00
6dcf79fc27
[Bug fix][metrics] correct fe collector type for jvm_gc ( #28784 )
...
Co-authored-by: 胥剑旭 <xujianxu@xujianxudeMacBook-Pro.local >
2023-12-21 16:21:13 +08:00
4ddef31c20
(fix)[meta][export] fix replay export NPE issue ( #28752 )
...
The ConnectionContext does not exist in replay thread
2023-12-21 14:39:00 +08:00
5d8c465644
[regression](p2) fix test cases result ( #28768 )
...
regression-test/data/external_table_p2/hive/test_hive_hudi.out
regression-test/data/external_table_p2/hive/test_hive_to_array.out
regression-test/suites/external_table_p2/tvf/test_local_tvf_compression.groovy
regression-test/suites/external_table_p2/tvf/test_path_partition_keys.groovy
regression-test/data/external_table_p2/hive/test_hive_text_complex_type.out
2023-12-21 14:38:30 +08:00
4ee661202e
[improve](transaction) extend abort transaction time ( #28662 )
2023-12-21 14:01:05 +08:00
d11bb11592
[enhancement](backup/restore) support alter s3 repo info about ak/sk/token ( #27027 )
...
in some cases:
s3.session_token/AWS_TOKEN will be expired after hours, and may be upload snapshot job will failed if data is big large;
as same reason, repo will be expired too when RepositoryMgr exec repo ping
so it need support alter s3 repo properties about ak/sk/token and update upload snapshot job properties to continue backup.
Signed-off-by: nextdreamblue <zxw520blue1@163.com >
2023-12-21 13:21:55 +08:00
223454d1db
[feature](Nereids) elimite inner join by foreign key ( #28486 )
2023-12-21 12:58:55 +08:00
db523dafcb
[improve](move-memtable) limit task num in load stream flush token ( #28748 )
2023-12-21 12:19:58 +08:00
425bbaa4fd
[test](Nereids): add assert debug log for TopnToMaxTest ( #28755 )
2023-12-21 12:01:15 +08:00
e9848066c9
[FIX](type) fix matchExactType for complex type ( #28233 )
...
fe matchExactType function should call type.matchTypes for its own logic, do not switch case to do special logic otherwise we may meet core in be like this.
```
F20231208 18:54:39.359673 680131 block.h:535] Check failed: _data_types[i]->is_nullable() target type: Struct(l_info:Nullable(Array(Nullable(String)))) src type: Struct(col:Nullable(Array(Nullable(UInt8))))
*** Check failure stack trace: ***
@ 0x5584e952b926 google::LogMessage::SendToLog()
@ 0x5584e9527ef0 google::LogMessage::Flush()
@ 0x5584e952c169 google::LogMessageFatal::~LogMessageFatal()
@ 0x5584cf17201e doris::vectorized::MutableBlock::merge_impl<>()
@ 0x5584ceac4b1d doris::vectorized::MutableBlock::merge<>()
@ 0x5584d4dd7de3 doris::vectorized::VUnionNode::get_next_const()
@ 0x5584d4dd9a45 doris::vectorized::VUnionNode::get_next()
@ 0x5584bce469bd std::__invoke_impl<>()
@ 0x5584bce466d0 std::__invoke<>()
@ 0x5584bce465c7 _ZNSt5_BindIFMN5doris8ExecNodeEFNS0_6StatusEPNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS1_St12_PlaceholderILi1EESC_ILi2EESC_ILi3EEEE6__callIS2_JOS4_OS7_OS8_EJLm0ELm1ELm2ELm3EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
@ 0x5584bce46358 std::_Bind<>::operator()<>()
@ 0x5584bce46208 std::__invoke_impl<>()
@ 0x5584bce46178 _ZSt10__invoke_rIN5doris6StatusERSt5_BindIFMNS0_8ExecNodeEFS1_PNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS3_St12_PlaceholderILi1EESD_ILi2EESD_ILi3EEEEJS5_S8_S9_EENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESL_E4typeEOSM_DpOSN_
@ 0x5584bce45c18 std::_Function_handler<>::_M_invoke()
@ 0x5584bce6412f std::function<>::operator()()
@ 0x5584bce56382 doris::ExecNode::get_next_after_projects()
@ 0x5584bce26218 doris::PlanFragmentExecutor::get_vectorized_internal()
@ 0x5584bce2431b doris::PlanFragmentExecutor::open_vectorized_internal()
@ 0x5584bce22a96 doris::PlanFragmentExecutor::open()
@ 0x5584bce27c9d doris::PlanFragmentExecutor::execute()
@ 0x5584bcbdb3f8 doris::FragmentMgr::_exec_actual()
@ 0x5584bcbf982f doris::FragmentMgr::exec_plan_fragment()::$_0::operator()()
@ 0x5584bcbf9715 std::__invoke_impl<>()
@ 0x5584bcbf96b5 _ZSt10__invoke_rIvRZN5doris11FragmentMgr18exec_plan_fragmentERKNS0_23TExecPlanFragmentParamsERKSt8functionIFvPNS0_12RuntimeStateEPNS0_6StatusEEEE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESH_E4typeEOSI_DpOSJ_
@ 0x5584bcbf942d std::_Function_handler<>::_M_invoke()
@ 0x5584b9dfd883 std::function<>::operator()()
@ 0x5584bd6e3929 doris::FunctionRunnable::run()
@ 0x5584bd6cf8ce doris::ThreadPool::dispatch_thread()
```
2023-12-21 11:49:05 +08:00
12089cda08
[refactor](nereids) make NormalizeAggregate rule more clear and readable ( #28607 )
2023-12-21 11:42:22 +08:00
62833006d6
[test](Nereids): remove shape plan project and distribute in eager test ( #28724 )
2023-12-21 11:14:26 +08:00
34fd376f33
[fix](publish version) fix publish fail but return ok ( #28425 )
2023-12-21 11:10:08 +08:00
9d1a0b5612
[improve](rpc) Log channel state before shutdown backend service client ( #28667 )
2023-12-21 10:40:56 +08:00
42c5dba9df
[fix](fe ut) fix PropertyConverterTest ( #28722 )
2023-12-21 10:39:54 +08:00
0e0ad3c4b1
[fix](regression) fix stream load properties case fail ( #28680 )
2023-12-21 10:39:19 +08:00
bcf2683b9d
[fix](scanner) fix concurrency bugs when scanner is stopped or finished ( #28650 )
...
`ScannerContext` will schedule scanners even after stopped, and confused with `_is_finished` and `_should_stop`.
Only Fix the concurrency bugs when scanner is stopped or finished reported in https://github.com/apache/doris/pull/28384
2023-12-21 10:37:58 +08:00
970e1c8475
[fix](group_commit) fix group commit cancel stuck ( #28749 )
2023-12-21 10:32:21 +08:00
5bf77c8f74
[Fix] (schema change) forbid adding time type column ( #28751 )
2023-12-21 10:27:50 +08:00
e4a58b7407
[Feature](materialized-view) support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg ( #28747 )
...
support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg
2023-12-21 10:27:33 +08:00
007f152e5e
[Improve](compile) add __AVX2__ macro for JsonbParser ( #28754 )
...
* [Improve](compile) add `__AVX2__` macro for JsonbParser
* throw exception instead of CHECK
2023-12-21 10:25:26 +08:00
8759bce426
[fix](stmt):fix CreateTableStmt toSql placed comment in wrong place ( #27504 )
...
Issue Number: close #27474
Co-authored-by: tongyang.han <tongyang.han@jiduauto.com >
2023-12-21 09:57:20 +08:00
a443a39e2c
[enhance](blacklist) seperate blacklist conf from heartbeat ( #28638 )
...
There is a circuit breaker lasting for 2 minutes in grpc, then if a be is down and up again, send fragments to the be fails lasting for 2 minutes.
2023-12-21 00:17:45 +08:00
18ad8562f2
[refactor](broadcastbuffer) using a queue to remove ref and unref codes ( #28698 )
...
Co-authored-by: yiguolei <yiguolei@gmail.com >Add a new class broadcastbufferholderqueue to manage holders
Using shared ptr to manage holders, not use ref and unref, it is too difficult to maintain.
2023-12-20 21:23:25 +08:00
a8dcca98ec
[FIX](explode)fix explode array decimal ( #28744 )
...
* fix explode with array<decimal> has specific precision at old planner
2023-12-20 20:19:56 +08:00
280a01b815
[pipelineX](improvement) Support global runtime filter ( #28692 )
2023-12-20 20:06:26 +08:00
504693be7f
[bug](coredump) Fix coredump in aggregation node's destruction( #28684 )
...
fix coredump in aggregation node's destruction
2023-12-20 20:02:48 +08:00
36857006cd
[Fix](json reader) fix json reader crash due to fmt::format_to ( #28737 )
...
```
4# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
5# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
6# 0x00005622F33D22B1 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
7# 0x00005622F33D2404 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
8# fmt::v7::detail::error_handler::on_error(char const*) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
9# char const* fmt::v7::detail::parse_replacement_field<char, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&>(char const*, char const*, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
10# void fmt::v7::detail::vformat_to<char>(fmt::v7::detail::buffer<char>&, fmt::v7::basic_string_view<char>, fmt::v7::basic_format_args<fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<fmt::v7::type_identity<char>::type>, fmt::v7::type_identity<char>::type> >, fmt::v7::detail::locale_ref) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
11# doris::vectorized::NewJsonReader::_append_error_msg(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:924
12# doris::vectorized::NewJsonReader::_set_column_value
```
2023-12-20 19:58:30 +08:00
ea646c6682
[fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE ( #28723 )
...
when replay addTaskResult log,will create one ConnectContext,and set Env.getCurrentEnv,then store this ctx in ConnectContext.threadLocalInfo,threadLocalInfo is static,so this ctx can not be recycling，Env of replay thread also can not be recycling
2023-12-20 19:46:58 +08:00
7b96730e87
[fix](block) fix nullptr in MutableBlock::allocated_bytes ( #28738 )
2023-12-20 19:46:13 +08:00
49eed98c1e
[fix](tvf)Fixed the avro-scanner projection pushdown failing to query on multiple BEs ( #28709 )
2023-12-20 19:39:26 +08:00
40afee0097
[fix](Nereids) delete partition failed ( #28717 )
...
1. parser's partitionSpec changed unexpectly by PR #26492
2. delete without using should support un-equals expression
2023-12-20 19:31:53 +08:00
