8ab63a9434
[regression-test][memtable] test memtable flush is high priority for vtable writerV2 ( #28503 )
2023-12-16 22:29:06 +08:00
8c05f7a784
[refactor](cluster)(step-4) remove cluster related to Database ( #27861 )
...
Issue Number: #19897
Remove `default_cluster` prefix related to database.
When upgrading, all prefix will be removed.
2023-12-16 18:28:53 +08:00
608baae001
[docker][regregression]update routine load cases #28450
...
Co-authored-by: 胥剑旭 <xujianxu@xujianxudeMacBook-Pro.local >
2023-12-16 17:57:39 +08:00
ad8faedac4
[fix](txn_manager) Fix wrong use of std::map::erase in TxnManager::delete_txn #28507
2023-12-16 14:50:46 +08:00
a3e2c6affe
[fix](jdbc catalog) fix JdbcScanNode NOT CompoundPredicate filter expr handling errors ( #28497 )
2023-12-16 12:54:55 +08:00
92a4a9770c
[improvement](hint) query fail print tablet detail info ( #28476 )
2023-12-16 12:54:25 +08:00
b11b76e778
[fix](full compaction) Full compaction should hold meta lock when modifying tablet's meta data ( #28449 )
2023-12-16 12:37:29 +08:00
469edbdd3d
[feature](executor)make scan task wait timeout config #28467
2023-12-16 11:36:15 +08:00
920c75c870
[fix](ci)tpch pipeline add check ( #28370 )
...
Co-authored-by: stephen <hello-stephen@qq.com >
2023-12-16 11:11:28 +08:00
3ea68d576b
[improve](group commit) Fix select tablet policy for random partition and remove some log ( #28498 )
...
This pr contains 2 improvements:
For random partition table, select tablet in the original way for load balance;
Skip execute stmt audit log since it's expensive in CPU;
2023-12-16 11:02:52 +08:00
f12a225844
[fix](session variables) Make default value of max_execution_time same to query_timeout #28474
...
Current problem, UNSET global VARIABLE ALL will write an oplog, which makes query_timeout = 0 when we replay it in a future time-stamp. So we change default value of max_execution_time to 90000 which is consistent to query_timeout default value.
2023-12-16 10:59:05 +08:00
4538f1ba8f
[feature](pipelineX) add local_shuffle in nested loop join #28428
2023-12-16 10:53:13 +08:00
f741ce5b7b
[fix](iterator) Fix mem leak when initial iterator failed ( #28480 )
2023-12-16 10:49:05 +08:00
f770403cca
[enhancement](pipeline) add bvar for pipeline fragment instance and task ( #28500 )
2023-12-16 10:47:53 +08:00
20d815f0e7
[refactor](style) Using C++style and changing to smart pointers ( #28454 )
2023-12-16 10:44:43 +08:00
fb925bdd08
[Bug](memory) Fix exception-unsafe in aggregation node ( #28483 )
...
The alloc function may throw std::bad_alloc exception when the process memory exceed limit.
be.INFO:
W1214 09:14:17.434849 771103 mem_tracker_limiter.cpp:204] Memory limit exceeded:<consuming tracker:<Load#Id=28448230da1f432e-8a66597e1032
9235>, process memory used 20.41 GB exceed limit 18.76 GB or sys mem available 9.04 GB less than low water mark 1.60 GB, failed alloc siz
e 1.86 MB>, executing msg:<execute:<>>. backend xx.x.x.xxx process memory used 20.41 GB, limit 18.76 GB. If query tracker exceed, set ex ec_mem_limit=8G to change limit, details see be.INFO.
Process Memory Summary:
OS physical memory 31.26 GB. Process memory usage 20.41 GB, limit 18.76 GB, soft limit 16.88 GB. Sys available memory 9.04 GB, low wa
ter mark 1.60 GB, warning water mark 3.20 GB. Refresh interval memory growth 0 B
Alloc Stacktrace:
@ 0x555cd858bee9 doris::MemTrackerLimiter::print_log_usage()
@ 0x555cd859a384 doris::ThreadMemTrackerMgr::exceeded()
@ 0x555cd85a0ac4 malloc
@ 0x555cd8fcf368 Allocator<>::alloc()
@ 0x555cd8fdbdaf doris::vectorized::Arena::add_chunk()
@ 0x555cd96dc0ab doris::vectorized::AggregateDataContainer::_expand()
@ 0x555cd96aded8 (unknown)
@ 0x555cd969fa2c doris::vectorized::AggregationNode::_pre_agg_with_serialized_key()
@ 0x555cd96d1d61 std::_Function_handler<>::_M_invoke()
@ 0x555cd967ab0b doris::vectorized::AggregationNode::get_next()
@ 0x555cd81282a6 doris::ExecNode::get_next_after_projects()
@ 0x555cd8452968 doris::PlanFragmentExecutor::get_vectorized_internal()
@ 0x555cd845553b doris::PlanFragmentExecutor::open_vectorized_internal()
@ 0x555cd8456a9e doris::PlanFragmentExecutor::open()
@ 0x555cd842f200 doris::FragmentExecState::execute()
@ 0x555cd843280e doris::FragmentMgr::_exec_actual()
@ 0x555cd8432d42 _ZNSt17_Function_handlerIFvvEZN5doris11FragmentMgr18exec_plan_fragmentERKNS1_23TExecPlanFragmentParamsESt8funct
ionIFvPNS1_20PlanFragmentExecutorEEEEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x555cd86ead05 doris::ThreadPool::dispatch_thread()
@ 0x555cd86e015f doris::Thread::supervise_thread()
@ 0x7f3321593ea5 start_thread
@ 0x7f33218a69fd __clone
@ (nil) (unknown)
2023-12-15 19:17:18 +08:00
0f93ee8793
[fix](Nereids): TransposeSemiJoinAgg can't apply in Scalar Agg ( #28434 )
...
Scalar Agg shouldn't be pushdown, it will cause wrong result
2023-12-15 16:18:16 +08:00
8986bb6bb4
[fix](Planner): parse more Punctuation Date/DateTime ( #28432 )
...
parse more Punctuation as separator, like `2021@01@01 00/00/00`;
2023-12-15 16:17:44 +08:00
0f25a4b3c6
[bug](json)Fix the problem of be down caused by json path ending with \ ( #28180 )
2023-12-15 15:57:08 +08:00
088bb80a9c
[fix](test) fix case of test_unique_table_new_sequence ( #28442 )
...
pr #28105 and #28031 merging conflict caused this case to fail.
2023-12-15 15:10:39 +08:00
501a79a45c
[Fix](format) compatible run_clang_format format string with python2/3.6/higher ( #28469 )
2023-12-15 15:03:23 +08:00
97b033813a
[perf](Nereids) add back canEliminate temporary ( #28017 )
2023-12-15 14:26:29 +08:00
2018ab23f0
[chore](build) Add MVN_OPT env variable to enrich building FE with extra options ( #28375 )
...
e.g. just export or add it to custom_env.sh
```
export MVN_OPT="-o"
```
will build FE with maven option "-o" (offline), which means maven does
not need to download meta from maven repo, it is useful for saving time
if the internet is unstable or unusable.
2023-12-15 13:20:39 +08:00
e6b135c76a
[improvement](fe) Add reason log when Env is not ready ( #28286 )
2023-12-15 12:22:06 +08:00
6f3fb81965
[fix](doc) spell errors fixes multi-tenant.md ( #28436 )
2023-12-15 12:21:46 +08:00
4c51558f6b
[feature](nereids) Support basic aggregate rewrite and function rollup using materialized view ( #28269 )
...
Add aggregate materializedviewRules for query rewrite.
it support the query rewrite as following:
def mv = "select lineitem.L_LINENUMBER, orders.O_CUSTKEY, sum(O_TOTALPRICE) as sum_alias " +
"from lineitem " +
"inner join orders on lineitem.L_ORDERKEY = orders.O_ORDERKEY " +
"group by lineitem.L_LINENUMBER, orders.O_CUSTKEY "
def query = "select lineitem.L_LINENUMBER, sum(O_TOTALPRICE) as sum_alias " +
"from lineitem " +
"inner join orders on lineitem.L_ORDERKEY = orders.O_ORDERKEY " +
"group by lineitem.L_LINENUMBER"
2023-12-15 11:30:02 +08:00
c4242ab69e
[Chore](Job)Add the configuration of the maximum number of persistence tasks for the job ( #28411 )
2023-12-15 11:14:06 +08:00
8661b5ec21
[Bug](function) fix npe on select http_stream directly ( #28423 )
...
fix npe on select http_stream directly
2023-12-15 11:09:45 +08:00
ce60064573
[regression-test](variant) fix unstable query result m… ( #28390 )
...
* [regression-test](variant) fix unstable query result for unique key model
* fix p2 case
* add msg
2023-12-15 10:52:50 +08:00
1877389f12
[fix](Nereids) set card to olap table break card block rule ( #28417 )
...
we have card block rule to avoid scan too many data.
so we must set olap scan card by only scanned bucket.
2023-12-15 10:28:05 +08:00
4d9b6c272d
[Fix](vcompound pred) Corrected evaluation for compound predicates with constant columns ( #28421 )
2023-12-15 10:10:48 +08:00
eb99e4270d
[Fix](parquet_reader) Fix dict filtering doesn't work with plain dict encoding in parquet reader. ( #28290 )
2023-12-15 09:27:02 +08:00
eebedbc879
[optimize](cooldown)Reduce unnecessary sort operations for vector ( #27147 )
...
Co-authored-by: xingying01 <xingying01@corp.netease.com >
2023-12-15 00:13:56 +08:00
415c6d854d
[fix](partial update) Fix some bugs about partial update ( #28358 )
2023-12-15 00:04:29 +08:00
8ca7bd8f98
[enhancement](bitmap)support bitmap type for non-key column in duplicate table ( #28392 )
2023-12-14 23:59:12 +08:00
310d1ab9a9
[feature](pipelineX)add parent-child relationship between the sink and downstream operators in profile ( #28406 )
2023-12-14 23:54:54 +08:00
4300fdce43
[feature](merge-on-write) enable merge-on-write by default again ( #28105 )
...
fix #27188 , #28096
2023-12-14 19:35:52 +08:00
a540b7eb43
[fix](Nereids) multi_distinct_sum should inherit NullableAggregateFunction ( #28349 )
2023-12-14 19:27:06 +08:00
82a91380e6
[enhancement](compaction) Add support for limiting low priority compaction scheduling ( #27648 )
2023-12-14 18:31:23 +08:00
429a3ed4df
[fix](Nereids) simplify range produce true when reference is nullable ( #28386 )
...
if reference is nullable, even if range is all, we should not return
true, but should return reference is not null. for example,
before simplify: c1 > 5 or c1 < 10
after simplify:
c1 is nullable: c1 IS NOT NULL
c1 is not nullable: TRUE
2023-12-14 18:10:17 +08:00
23941ef305
[improve](pipelineX) improve partition node dependency logical ( #28399 )
2023-12-14 18:05:39 +08:00
01c94a554d
[fix](autoinc) Fix broker load when target table has autoinc column ( #28402 )
2023-12-14 18:02:54 +08:00
c08ab9edc7
[feature](HiveCatalog) Support for getting hive meta data from relational databases under HMS ( #28188 )
2023-12-14 17:50:17 +08:00
38e79e32fa
[fix](mtmv)fix start time can not be earlier than the current time ( #28379 )
2023-12-14 17:28:04 +08:00
9fe2fce306
[minor](refactor) remove unused code ( #28383 )
2023-12-14 17:16:41 +08:00
6c9c1ef2f2
[test](regression) add flink sync db case ( #19328 )
...
add flink sync db case
Mock cdc sync data write
Contains multi-table import, add and drop columns
2023-12-14 17:06:17 +08:00
e1f7b6cff4
[typo](doc) update spark connector doc for version 1.3.0 ( #28239 )
...
Co-authored-by: gnehil <gnehil489@github>
2023-12-14 16:48:10 +08:00
634292ec1d
[Enhancement](build support) make run_clang_format compatible with python3 ( #28404 )
...
run_clang_format.py in python3 env will produce error like "ModuleNotFoundError: No module named 'distutils.util' ", try to fix this.
2023-12-14 16:02:47 +08:00
9e62e3e868
[doc](readme)Update README.md ( #28391 )
2023-12-14 14:50:43 +08:00
ee24667b9f
[fix](group commit) Fix some group commit problems ( #28319 )
2023-12-14 14:38:56 +08:00
