e962a7309b
[Chore](runtime-filter) adjust some check and error msg on runtime filter ( #35018 ) ( #35251 )
...
adjust some check and error msg on runtime filter
2024-05-23 11:20:02 +08:00
ced0093d74
[fix](mem_tracker] attach mem tracker in FragmentMgr::apply_filter ( #35128 )
2024-05-22 10:02:46 +08:00
2ed6a00fd1
[opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory ( #34985 ) ( #35070 )
2024-05-22 09:53:45 +08:00
42425808a1
[Cherry-Pick](branch-2.1) Pick "Fix multiple replica partial update auto inc data inconsistency problem #34788 " ( #35056 )
...
* [Fix](auto inc) Fix multiple replica partial update auto inc data inconsistency problem (#34788 )
* **Problem:** For tables with auto-increment columns, updating partial columns can cause data inconsistency among replicas.
**Cause:** Previously, the implementation for updating partial columns in tables with auto-increment columns was done independently on each BE (Backend), leading to potential inconsistencies in the auto-increment column values generated by each BE.
**Solution:** Before distributing blocks, determine if the update involves partial columns of a table with an auto-increment column. If so, add the auto-increment column to the last column of the block. After distributing to each BE, each BE will check if the data key for the partial column update exists. If it exists, the previous auto-increment column value is used; if not, the auto-increment column value from the last column of the block is used. This ensures that the auto-increment column values are consistent across different BEs.
* 2
* [Fix](regression-test) Fix auto inc partial update unstable regression test (#34940 )
2024-05-20 15:43:46 +08:00
dff6171546
[fix](auto inc) db_id and table_id should be int64_t instead of int32_t ( #34912 )
2024-05-18 18:29:59 +08:00
1a24895257
[opt](routine-load) optimize routine load task thread pool and related param( #32282 ) ( #34896 )
2024-05-15 12:42:02 +08:00
0ae1b9c70a
[chore](remove code) Remove dragonbox related ( #34528 )
...
* Revert "[refactor](mysql result format) use new serde framework to tuple convert (#25006 )"
This reverts commit e5ef0aa6d439c3f9b1f1fe5bc89c9ea6a71d4019.
* run buildall
* MORE
* FIX
2024-05-13 22:16:57 +08:00
cdc950f2c3
[improvement](spill) improve spill log printing
2024-05-12 19:33:27 +08:00
7a172a55ab
[fix](memory) Fix prune all LRU Cache based on number #34601 ( #34736 )
2024-05-12 11:47:53 +08:00
0a79c547ff
[Refactor](Sink) Remove is_append mode in table sink ( #34684 )
...
Remove the is_append mode from the sink component due to the following reasons:
1. The performance improvement from this mode is relatively minor, approximately 10%, as demonstrated in previous benchmarks.
2. The mode complicates maintenance. It requires a separate data writing path to avoid copying, which increases complexity and poses a risk of potential data loss.
I've already test the compability with previous version
2024-05-11 11:20:10 +08:00
7ba66c5890
[branch-2.1](routine-load) do not schedule task when there is no data ( #34654 )
2024-05-11 11:01:18 +08:00
853dbdcb00
[Feature](PreparedStatement) implement general server side prepared ( #33807 )
2024-05-10 22:10:11 +08:00
093fe354c8
[Improve](cache) Estimated column reader memory to control segment cache ( #34526 )
2024-05-10 22:05:20 +08:00
6c30913a31
[bugfix](memtracker) should count cancelled query in wg memory ( #34443 )
...
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-05-10 14:35:28 +08:00
dec5f0ca98
Revert "[fix](profile) Fix reporting the profile while building the p… ( #34498 )
...
* Revert "[fix](profile) Fix reporting the profile while building the pipeline profile. (#34215 )"
This reverts commit eb0d963389e1b7d150cbc18c927091648e0a60f7.
* Revert "[feature](profile) sort pipelineX task by total time #34053 "
This reverts commit 67b394f2b0dddab3801d2faa82a91c52ef875e76.
2024-05-07 22:58:50 +08:00
a81beb19c2
[fix](load) fix repeatedly open tablets_channel when tablets_channel already cancelled ( #34442 )
2024-05-06 23:15:33 +08:00
11ca738261
[fix](memory) Fix thread context init in MacOS and not use memory tracker ( #34125 )
2024-05-06 20:11:20 +08:00
ab5ee81811
[fix](memory) Fix page cache memory tracker consumption in prune ( #34320 )
2024-05-06 12:53:11 +08:00
7248420cfd
[chore](session_variable) Add 'data_queue_max_blocks' to prevent the DataQueue from occupying too much memory. ( #34017 ) ( #34395 )
2024-05-05 21:20:33 +08:00
8abd136ba2
[Improvement](executor)Refactor Workload group memory GC ( #33797 )
...
* just gc group's overcommit query when minor gc
* add process usage
2024-04-30 19:34:31 +08:00
7d77fd0286
[fix](profile) Fix reporting the profile while building the pipeline profile. ( #34215 ) ( #34326 )
2024-04-30 11:38:03 +08:00
cd1c9edd71
[fix](pipeline-load) fix no error url when data quality error and total rows is negative ( #34072 ) ( #34204 )
...
Co-authored-by: HHoflittlefish777 <77738092+HHoflittlefish777@users.noreply.github.com >
2024-04-27 18:19:08 +08:00
47ded2c6a0
Revert "[fix](compile) fix two compile errors on MacOS ( #33834 ) ( #34005 )"
...
This reverts commit 743fb62a2c42cc5cc662583c235f7336d5e6ddef.
2024-04-26 00:55:21 +08:00
743fb62a2c
[fix](compile) fix two compile errors on MacOS ( #33834 ) ( #34005 )
2024-04-25 19:39:35 +08:00
47b54d4bd5
Fix remote scan pool ( #33976 )
2024-04-25 15:04:43 +08:00
5f2d0e3d53
[Fix](executor)Fix when Fe send empty wg list to be may cause query failed. ( #34074 )
2024-04-25 12:01:44 +08:00
67b394f2b0
[feature](profile) sort pipelineX task by total time #34053
2024-04-25 12:01:44 +08:00
f6ec64c6ad
[fix](exception) Fix Block noexcept method not throw exception ( #34002 )
2024-04-24 17:13:50 +08:00
9bb149b3be
[fix](stream-load) fix query id is zero in stream load log ( #33954 )
2024-04-22 22:33:24 +08:00
615765c1c0
[improvement](spill) improve spill directory and fix bugs ( #33900 )
...
* [improvement](spill) improve spill directory and fix bugs
* fix
2024-04-22 11:28:22 +08:00
09b973db49
[Chore](runtime-filter) adjust need_local_merge setting conditions ( #33886 )
2024-04-19 23:50:04 +08:00
25358564ca
[Fix](compile) Fix gcc compile on master ( #33864 )
...
This is imported by #33511 . wrongly used
ColumnStr<T> ();
which violate C++20 standard(see https://wg21.cmeerw.net/cwg/issue2237 ) but still supported by clang up until now(see llvm/llvm-project#58112 )
2024-04-19 23:41:37 +08:00
c747714c18
[fix](memory) Fix ExecEnv destroy memory tracking ( #33781 )
...
disable memory tracking when ExecEnv destroy.
fix memory tracker label convert to query id
2024-04-19 15:03:10 +08:00
8b061c7055
[Enhancement](group commit) Add fault injection case for group commit
2024-04-19 15:03:06 +08:00
ba05ef4405
[Chore](runtime-filter) add tmp debug info to investigate unknown filter error #33857
2024-04-18 21:03:09 +08:00
1300317723
[Exec](join) Support column string64 to avoid join failed in string size overflow the uint32 ( #33511 ) ( #33850 )
2024-04-18 19:43:08 +08:00
20b37e7a18
Add workload group id in workload policy's property ( #33483 )
2024-04-17 23:42:14 +08:00
9bcb23351c
[Improvement](runtime-filter) make sync rf size work when need_local_merge ( #33717 )
...
make sync rf size work when need_local_merge
2024-04-17 23:42:14 +08:00
07a8f44443
[improvement](spill) improve config and fix spill bugs ( #33519 )
2024-04-17 23:42:13 +08:00
e0ec2da29b
[fix](routine-load) fix get kafka offset timeout may too long ( #33502 )
2024-04-17 23:42:12 +08:00
e85a2c8866
[Chore](status) change unknow filter error to internal error ( #33633 )
2024-04-17 23:42:12 +08:00
8ee8de7857
[Fix](executor)reset remote scan thread num #33579
2024-04-17 23:42:11 +08:00
01f333086d
[pipelineX](fix) Fix data pooling judgement for bucket join ( #33533 )
2024-04-17 23:42:00 +08:00
b2b385a4ff
[improve](fold) support complex type for constant folding ( #32867 )
2024-04-17 23:41:59 +08:00
fefbde8927
[log](move-memtable) improve logs in vtablet_writer_v2 and load_stream ( #33103 )
2024-04-12 15:09:25 +08:00
a4924dabb7
[enhancement](exception) enble exception logic in pipeline execute thread ( #33437 )
...
* [enhancement](exception) enble exception logic in pipeline execute thread
* f
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-04-12 15:09:25 +08:00
5f30463bb3
[Chore](descriptors) remove unused codes for descriptors ( #33408 )
...
remove unused codes for descriptors
2024-04-12 15:09:25 +08:00
e841d82ffb
[Enhancement](hive-writer) Adjust table sink exchange rebalancer params. ( #33397 )
...
Issue Number: #31442
Change table sink exchange rebalancer params to node level and adjust these params to improve write performance by better balance.
rebalancer params:
```
DEFINE_mInt64(table_sink_partition_write_min_data_processed_rebalance_threshold,
"26214400"); // 25MB
// Minimum partition data processed to rebalance writers in exchange when partition writing
DEFINE_mInt64(table_sink_partition_write_min_partition_data_processed_rebalance_threshold,
"15728640"); // 15MB
```
2024-04-12 13:09:56 +08:00
3d66723214
[branch-2.1](auto-partition) pick auto partition and some more prs ( #33523 )
2024-04-11 17:12:17 +08:00
5688c28364
[Bug](runtime-filter) try to fix heap use after free on runtime filter send filter size ( #33465 ) ( #33522 )
2024-04-11 13:10:24 +08:00
