18ad8562f2
[refactor](broadcastbuffer) using a queue to remove ref and unref codes ( #28698 )
...
Co-authored-by: yiguolei <yiguolei@gmail.com >Add a new class broadcastbufferholderqueue to manage holders
Using shared ptr to manage holders, not use ref and unref, it is too difficult to maintain.
2023-12-20 21:23:25 +08:00
280a01b815
[pipelineX](improvement) Support global runtime filter ( #28692 )
2023-12-20 20:06:26 +08:00
fe184e322a
[code](pipelineX) refine some pipelineX code ( #28570 )
2023-12-20 11:45:06 +08:00
c26c0c31a5
[refactor](runtimefilter) do not use QueryContext in runtime filter ( #28559 )
2023-12-20 10:28:55 +08:00
9434ee5710
[fix](load) fix memtracking orphan too large ( #28600 )
2023-12-19 12:41:19 +08:00
d7dd7b775b
enhance performance for broken tablet checking under multi-core scenario with a coarse-grained read lock ( #28552 )
2023-12-19 12:33:34 +08:00
66fbb22ad7
[fix](group commit) Fix some wal problems on group commit ( #28554 )
2023-12-19 09:51:03 +08:00
73f7b61019
[refactor](scanner) use weak ptr to lock task execution context to avoid core in scanner dctor ( #28493 )
...
using weak ptr as a lock between fragment execute thread and scanner thread, to solve the core problem in scanner's dctor to access scannode's profile.
2023-12-18 14:09:32 +08:00
61de49c727
[case](regression) Test duplicated load id ( #28251 )
...
Co-authored-by: qinhao <qinhao@newland.com.cn >
2023-12-16 22:41:51 +08:00
f770403cca
[enhancement](pipeline) add bvar for pipeline fragment instance and task ( #28500 )
2023-12-16 10:47:53 +08:00
310d1ab9a9
[feature](pipelineX)add parent-child relationship between the sink and downstream operators in profile ( #28406 )
2023-12-14 23:54:54 +08:00
c00dca70e6
[pipelineX](local shuffle) Support parallel execution despite of tablet number ( #28266 )
2023-12-14 12:53:54 +08:00
a719d7a222
[fix](memory) Fix LRU Cache of type NUMBER charge ( #28175 )
2023-12-13 11:15:57 +08:00
45b2dbab6a
[improve](group commit) Group commit support max filter ratio when rows is less than value in config ( #28139 )
2023-12-12 16:33:36 +08:00
d4f89389e3
[improve](group commit) Group commit support skip wal ( #27957 )
2023-12-11 19:38:32 +08:00
877935442f
[feature](pipelineX)use markFragments instead of markInstances in pipelineX ( #27829 )
2023-12-11 17:59:53 +08:00
cff1de29ce
[fix](group commit) Fix group commit memory calculation ( #28242 )
2023-12-11 17:05:26 +08:00
1e5ff40e17
[refactor](group commit) remove future block ( #27720 )
...
Co-authored-by: huanghaibin <284824253@qq.com >
2023-12-11 08:41:51 +08:00
4e86f9bab5
[improve](move-memtable) include and check offset when append data ( #28159 )
2023-12-09 16:21:36 +08:00
6da36e1077
[feature](merge-cloud) Refactor write path code by abstract base class ( #26537 )
...
Refactor write path code by abstract base class. Whether to use `StorageEngine` or `CloudStorageEngine` will be determined during compilation instead of runtime `config::cloud_mode` to avoid unexpected null pointer or undefined behavior issues caused by merging code.
Class that depend on `StorageEngine` but are shared by the cloud mode need to have an abstract base class. Common code should be extracted into the base class, while the code that depends on `StorageEngine` should be implemented in a `StorageEngine` mix-in class of the base class.
2023-12-08 14:50:36 +08:00
9c63dfd692
[improve](move-memtable) support variant in move-memtable ( #28084 )
...
* [improve](move-memtable) support variant in move-memtable
Pass the flush schema to the destination end of the sink.
2023-12-07 16:41:38 +08:00
c5f00fa746
[fix](move-memtable) temporary disable load stream profile ( #28029 )
2023-12-07 12:05:29 +08:00
bd808ae3b4
[cleanup](load) remove unused LoadChannel::get_tablets_channels ( #27981 )
2023-12-06 20:50:26 +08:00
3595f21405
[improvement](executor)clear unused cgroup path ( #27798 )
...
* clear unused cgroup path
* use C++ api
* add gcc header
2023-12-05 14:18:23 +08:00
54fe1a166b
[Refactor](scan) refactor scan scheduler to improve performance ( #27948 )
...
* [Refactor](scan) refactor scan scheduler to improve performance
* fix pipeline x core
2023-12-05 13:03:16 +08:00
17016b9797
[improvement](decimal) use new way for decimal arithmetic precision promotion ( #27787 )
...
* [DNM](decimal) use new way for decimal arithmetic precision promotion
* [improvement](decimal) [DNM](decimal) use new way for decimal arithmetic precision promotion
1. [DNM](decimal) use new way for decimal arithmetic precision promotion
2. throw exception if it overflows for decimal arithmetics
3. throw exception if it overflows when casting among number types
* fix compile error of gcc
* improvement
---------
Co-authored-by: morrySnow <morrysnow@126.com >
2023-12-05 12:54:40 +08:00
75d0beb8cc
[fix](move-memtable) only report load stream profile in the end ( #27983 )
2023-12-05 11:30:54 +08:00
bd9db7423b
[fix](move-memtable) free resources before storage engine stop ( #27980 )
2023-12-05 11:15:05 +08:00
e3d2425d47
[Improvement](join) remove insert_indices_from_join and special judge for -1 ( #27779 )
...
remove insert_indices_from_join and special judge for -1
2023-12-04 11:03:22 +08:00
be30bd1e40
[improvement](spinlock) remove some potential bad spinlock usage ( #27904 )
...
* [improvement](spinlock) remove some potential spinlock usage
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2023-12-02 20:33:54 +08:00
421ab56c3e
[pipelineX](improvement) Support local shuffle for join and agg ( #27852 )
2023-12-02 20:17:18 +08:00
a1a75fcfbd
[fix](runtime filter) Fix extremely high CPU usage caused by rf merge #27894
2023-12-02 07:40:52 +08:00
1706699e7e
[fix](multi-catalog)support the max compute partition prune ( #27154 )
...
1. max compute partition prune,
we just support filter mc partitions by '='，it can filter just one partition
to support multiple partition filter and range operator('>','<', '>='..), the partition prune should be supported.
2. add max compute row count cache and partitionValues cache
3. add max compute regression case
2023-12-01 22:28:26 +08:00
0b7becd4b7
[fix](executor)Fix memtracker not set to task group #27699
2023-11-30 22:35:51 +08:00
5a4948f0f9
[fix](load) fix DataSink prepared check in PlanFragmentExecutor ( #27735 )
2023-11-30 15:24:04 +08:00
34e53acaea
[pipelineX](fix) Fix local exchange on pipelineX engine ( #27763 )
2023-11-30 11:16:20 +08:00
f1e9e6dba8
[fix](pipelineX) make RuntimeFilterTimerQueue graceful exit ( #27653 )
...
make RuntimeFilterTimerQueue graceful exit
2023-11-29 18:53:13 +08:00
d9d5468621
[feature](audit-log) add audit-log in insert into ( #27641 )
2023-11-29 15:01:57 +08:00
b93dd1d5f7
[enhancement](load) improve error msg for load when cancelled by mem gc ( #26809 )
...
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com >
2023-11-28 17:36:11 +08:00
f565f60bc3
[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default ( #27587 )
2023-11-28 13:02:30 +08:00
b48c40ed31
Make blockschduler first stop then delete ( #27645 )
2023-11-28 10:09:15 +08:00
13b26ee920
[Fix](core) Fix wal space back pressure core and add regression test ( #27311 )
2023-11-27 15:10:26 +08:00
3838b6fbae
[refine](pipelineX) refine some code in pipelineX ( #27472 )
2023-11-27 11:04:16 +08:00
d0fea8db27
[chore][log] Opt log, revert some log introduced by #25739 ( #26365 )
2023-11-27 10:48:02 +08:00
f3204ec18c
[fix](memory) Fix thread_context check failed: thread_mem_tracker()->label() == "Orphan" #27468
2023-11-24 20:57:41 +08:00
89d5efac46
[pipelineX](dependency) Use dependency to replace ready_to_execute ( #27528 )
2023-11-24 16:03:14 +08:00
2ea33518b0
[Opt](load) use batching to optimize auto partition ( #26915 )
...
use batching to optimize auto partition
2023-11-23 19:12:28 +08:00
b457856bd2
[chore](be) remove bthread scanner related codes ( #27417 )
2023-11-23 15:18:49 +08:00
4fc638c98c
[fix](memory) Fix make_top_consumption_snapshots heap-use-after-free #27434
2023-11-23 13:59:53 +08:00
75d1594d11
[pipelineX](minor) remove unused code ( #27446 )
2023-11-23 11:53:06 +08:00
