6dcf79fc27
[Bug fix][metrics] correct fe collector type for jvm_gc ( #28784 )
...
Co-authored-by: 胥剑旭 <xujianxu@xujianxudeMacBook-Pro.local >
2023-12-21 16:21:13 +08:00
4ddef31c20
(fix)[meta][export] fix replay export NPE issue ( #28752 )
...
The ConnectionContext does not exist in replay thread
2023-12-21 14:39:00 +08:00
4ee661202e
[improve](transaction) extend abort transaction time ( #28662 )
2023-12-21 14:01:05 +08:00
d11bb11592
[enhancement](backup/restore) support alter s3 repo info about ak/sk/token ( #27027 )
...
in some cases:
s3.session_token/AWS_TOKEN will be expired after hours, and may be upload snapshot job will failed if data is big large;
as same reason, repo will be expired too when RepositoryMgr exec repo ping
so it need support alter s3 repo properties about ak/sk/token and update upload snapshot job properties to continue backup.
Signed-off-by: nextdreamblue <zxw520blue1@163.com >
2023-12-21 13:21:55 +08:00
223454d1db
[feature](Nereids) elimite inner join by foreign key ( #28486 )
2023-12-21 12:58:55 +08:00
e9848066c9
[FIX](type) fix matchExactType for complex type ( #28233 )
...
fe matchExactType function should call type.matchTypes for its own logic, do not switch case to do special logic otherwise we may meet core in be like this.
```
F20231208 18:54:39.359673 680131 block.h:535] Check failed: _data_types[i]->is_nullable() target type: Struct(l_info:Nullable(Array(Nullable(String)))) src type: Struct(col:Nullable(Array(Nullable(UInt8))))
*** Check failure stack trace: ***
@ 0x5584e952b926 google::LogMessage::SendToLog()
@ 0x5584e9527ef0 google::LogMessage::Flush()
@ 0x5584e952c169 google::LogMessageFatal::~LogMessageFatal()
@ 0x5584cf17201e doris::vectorized::MutableBlock::merge_impl<>()
@ 0x5584ceac4b1d doris::vectorized::MutableBlock::merge<>()
@ 0x5584d4dd7de3 doris::vectorized::VUnionNode::get_next_const()
@ 0x5584d4dd9a45 doris::vectorized::VUnionNode::get_next()
@ 0x5584bce469bd std::__invoke_impl<>()
@ 0x5584bce466d0 std::__invoke<>()
@ 0x5584bce465c7 _ZNSt5_BindIFMN5doris8ExecNodeEFNS0_6StatusEPNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS1_St12_PlaceholderILi1EESC_ILi2EESC_ILi3EEEE6__callIS2_JOS4_OS7_OS8_EJLm0ELm1ELm2ELm3EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
@ 0x5584bce46358 std::_Bind<>::operator()<>()
@ 0x5584bce46208 std::__invoke_impl<>()
@ 0x5584bce46178 _ZSt10__invoke_rIN5doris6StatusERSt5_BindIFMNS0_8ExecNodeEFS1_PNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPS3_St12_PlaceholderILi1EESD_ILi2EESD_ILi3EEEEJS5_S8_S9_EENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESL_E4typeEOSM_DpOSN_
@ 0x5584bce45c18 std::_Function_handler<>::_M_invoke()
@ 0x5584bce6412f std::function<>::operator()()
@ 0x5584bce56382 doris::ExecNode::get_next_after_projects()
@ 0x5584bce26218 doris::PlanFragmentExecutor::get_vectorized_internal()
@ 0x5584bce2431b doris::PlanFragmentExecutor::open_vectorized_internal()
@ 0x5584bce22a96 doris::PlanFragmentExecutor::open()
@ 0x5584bce27c9d doris::PlanFragmentExecutor::execute()
@ 0x5584bcbdb3f8 doris::FragmentMgr::_exec_actual()
@ 0x5584bcbf982f doris::FragmentMgr::exec_plan_fragment()::$_0::operator()()
@ 0x5584bcbf9715 std::__invoke_impl<>()
@ 0x5584bcbf96b5 _ZSt10__invoke_rIvRZN5doris11FragmentMgr18exec_plan_fragmentERKNS0_23TExecPlanFragmentParamsERKSt8functionIFvPNS0_12RuntimeStateEPNS0_6StatusEEEE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESH_E4typeEOSI_DpOSJ_
@ 0x5584bcbf942d std::_Function_handler<>::_M_invoke()
@ 0x5584b9dfd883 std::function<>::operator()()
@ 0x5584bd6e3929 doris::FunctionRunnable::run()
@ 0x5584bd6cf8ce doris::ThreadPool::dispatch_thread()
```
2023-12-21 11:49:05 +08:00
12089cda08
[refactor](nereids) make NormalizeAggregate rule more clear and readable ( #28607 )
2023-12-21 11:42:22 +08:00
9d1a0b5612
[improve](rpc) Log channel state before shutdown backend service client ( #28667 )
2023-12-21 10:40:56 +08:00
42c5dba9df
[fix](fe ut) fix PropertyConverterTest ( #28722 )
2023-12-21 10:39:54 +08:00
5bf77c8f74
[Fix] (schema change) forbid adding time type column ( #28751 )
2023-12-21 10:27:50 +08:00
e4a58b7407
[Feature](materialized-view) support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg ( #28747 )
...
support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg
2023-12-21 10:27:33 +08:00
8759bce426
[fix](stmt):fix CreateTableStmt toSql placed comment in wrong place ( #27504 )
...
Issue Number: close #27474
Co-authored-by: tongyang.han <tongyang.han@jiduauto.com >
2023-12-21 09:57:20 +08:00
a443a39e2c
[enhance](blacklist) seperate blacklist conf from heartbeat ( #28638 )
...
There is a circuit breaker lasting for 2 minutes in grpc, then if a be is down and up again, send fragments to the be fails lasting for 2 minutes.
2023-12-21 00:17:45 +08:00
a8dcca98ec
[FIX](explode)fix explode array decimal ( #28744 )
...
* fix explode with array<decimal> has specific precision at old planner
2023-12-20 20:19:56 +08:00
280a01b815
[pipelineX](improvement) Support global runtime filter ( #28692 )
2023-12-20 20:06:26 +08:00
ea646c6682
[fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE ( #28723 )
...
when replay addTaskResult log,will create one ConnectContext,and set Env.getCurrentEnv,then store this ctx in ConnectContext.threadLocalInfo,threadLocalInfo is static,so this ctx can not be recycling，Env of replay thread also can not be recycling
2023-12-20 19:46:58 +08:00
49eed98c1e
[fix](tvf)Fixed the avro-scanner projection pushdown failing to query on multiple BEs ( #28709 )
2023-12-20 19:39:26 +08:00
40afee0097
[fix](Nereids) delete partition failed ( #28717 )
...
1. parser's partitionSpec changed unexpectly by PR #26492
2. delete without using should support un-equals expression
2023-12-20 19:31:53 +08:00
9a5ec43f05
[fix](nereids) Fix data wrong using mv rewrite and ignore case when getting mv related partition table ( #28699 )
...
1. Fix data wrong using mv rewrite
2. Ignore case when getting mv related partition table
3. Enable infer expression column name without alias when create mv
2023-12-20 17:59:46 +08:00
2b2d3d0eb1
[fix](meta_scanner) fix meta_scanner process ColumnNullable ( #28711 )
2023-12-20 17:41:38 +08:00
2683b1e8c3
[test](Nereids): add test for scalar agg ( #28712 )
2023-12-20 17:35:04 +08:00
08066a55f1
[Enhancement](auto-partition) change the behaviour when insert overwrite an auto partition table #28683
...
If we specific target partition(s) when inserting overwrite an auto partition table,
before:
could create new partition
now:
behalf just like non-auto partition table
2023-12-20 17:19:21 +08:00
552a5c1632
[fix](mtmv) fix insert overwrite getExecTimeout error ( #28700 )
...
should use InsertTimeout but QueryTimeout
2023-12-20 17:08:38 +08:00
9d5b9cc452
[fix](statistics)Fix drop stats fail silently bug. ( #28635 )
...
Drop stats use IN predicate to filter the column stats to delete. The default length of IN predicate is 1024, drop table stats with more than 1024 columns may fail.
This pr is to split the delete sql based on the IN predicate length.
2023-12-20 15:41:25 +08:00
bcc32b5b26
[feature](invert index) match_regexp feature added ( #28257 )
2023-12-20 14:30:35 +08:00
3e85797443
[fix](Nereids) should only do bind relation in view analyzer ( #28637 )
2023-12-20 10:47:51 +08:00
0ae87e705e
[fix](planner) ctas update datev1 to datev2 should use equals ( #28641 )
2023-12-20 10:31:49 +08:00
23784787b1
[fix](planner)should bind expr using no slot to correct tuple ( #28656 )
2023-12-20 10:29:59 +08:00
4c0080e237
[feat](Nereids) support outer join and aggregate bitmap rewrite by mv ( #28596 )
...
- Support left outer join rewrite by materialized view
- Support bitmap_union roll up to imp count(distinct)
- Support partition materialized view rewrite
2023-12-20 10:23:30 +08:00
044d7830c9
[improvement](transaction) reduce publish txn log ( #28277 )
2023-12-20 09:31:21 +08:00
9aa878ea1f
[Fix](Job)Fixed job scheduling missing certain time window schedules ( #28659 )
...
Since scheduling itself consumes a certain amount of time, the start time of the time window should not be the current time, but the end time of the last schedule.
2023-12-20 09:21:15 +08:00
fb3b0afb85
[opt](task-assignment) use consistent hash as default task assigner and cache the consistent hash ring ( #28522 )
...
1. Use consistent hash algo as the default assigner for file query scan node
A consistent assignment can better utilize the page cache of BE node.
2. Cache the consistent hash ring
Init a consistent hash ring is time-consuming because there a thousands of virtual node need to be added.
So cache it for better performance
2023-12-19 22:29:35 +08:00
d2275e73df
[Fix](transactional-hive) Fix hive transactional table return empty result. ( #28518 )
2023-12-19 19:38:48 +08:00
7da86c37ec
[fix](hive) add support for quoteChar and seperatorChar for hive ( #28613 )
...
add support for quoteChar and seperatorChar .
2023-12-19 19:35:03 +08:00
9c9249e911
[feature](inverted index) add ignore_above property to prevent long s… ( #28585 )
...
When string is too long, clucene will throw an error.
And the string is too long to analyze. So we ignore the string in index process when the string is longer than 256 bytes by default.
We add an poperty `ignore_above` for user to customize.
2023-12-19 18:54:36 +08:00
73a3d84c5e
[chore](user) Add user property parallel_fragment_exec_instance_num ( #28447 )
2023-12-19 18:33:01 +08:00
15e31d74e3
[feature](expr) Support kill query by query_id ( #28530 )
...
Issue Number: open #28517
2023-12-19 18:18:40 +08:00
71b7dcfb8f
[feature][executor]support workload schedule policy ( #28443 )
2023-12-19 18:00:02 +08:00
b99ac973d2
[fix](meta) update killed query state (#) ( #25917 )
2023-12-19 17:44:15 +08:00
a871409259
[feature](mtmv)after creating a materialized view, if other operations fail, roll back ( #28621 )
...
after create mtmv, if create job failed,need drop mtmv
2023-12-19 17:25:31 +08:00
111185407c
[Improve](tvf)jni-avro support split file ( #27933 )
2023-12-19 16:37:34 +08:00
372a2e20f9
[fix](Nereids) stats estimation of lessThan #28444
2023-12-19 15:45:03 +08:00
a104914880
[feature](mtmv)create mtmv support refresh_partition_num ( #28566 )
...
- create/alter mtmv support refresh_partition_num
- mtmv task according to refresh_partition_num executes refresh tasks in batches
- `tasks` tvf add column `CompletedPartitions` and `progress`
- fix mtmv can not `show temp partition` and `drop temp partition`
- fix task can not get error msg when insert overwrite error
- fix when the partition field is capitalized, the verification of creating a mtmv does not pass
2023-12-19 15:09:19 +08:00
87d36d0ffa
[opt](Nereids)when both Nereids and old parsers report errors, prompt error messages for the Nereids ( #28580 )
2023-12-19 12:04:08 +08:00
d24a1645d9
[improvement](http) add show_table_data http api ( #28380 )
...
In some cases, users need to get the data size of single replica of a table, and evaluate certain actions based on this, such as estimating the precise backup size.
Signed-off-by: nextdreamblue <zxw520blue1@163.com >
2023-12-19 11:44:33 +08:00
ddba98159e
[feature](Nereids) support datev1 and datetimev1 ( #28581 )
2023-12-19 11:31:28 +08:00
8da9bb659d
[feature](Nereids): eliminate semi join ( #28588 )
...
Eliminate Semi/Anti Join which is FALSE or TRUE.
2023-12-19 11:30:44 +08:00
88f0bee722
[Performance](point query)Opimize partition prune for point query ( #28150 )
...
* [Performance](point query)Opimize partition prune for point query
2023-12-19 11:00:13 +08:00
7b7845e695
[fix](planner) choice wrong length of string type output of union ( #28514 )
2023-12-19 10:43:39 +08:00
d6514618b2
[Improvement](decimal) reduce overhead on disable check decimal overflow ( #28249 )
...
reduce overhead on disable check decimal overflow
2023-12-19 10:12:30 +08:00
