Commit Graph

1346 Commits

Author SHA1 Message Date
747faeed17 [Enhancement](group commit) optimize some group commit code (#31392)
This PR optimizes some of the logic related to group commit:
1. Improved the error handling when there is insufficient WAL space during import.
2. Accounted for cases where the content length is negative during import.
3. Added missing error log printing in `group_commit_mgr.cpp`.
2024-02-28 13:05:57 +08:00
48804a978a [Fix](group commit) Fix group commit flink error message (#31350)
* When using stream processing frameworks like Flink with group commit mode enabled, the uncertain size of imported data makes such behavior prohibitive. Previously, to simplify the process, the error message for excessive data volume during streamload was combined with the one for group commit mode, leading to confusion for users when encountering errors indicating the data volume is too large during Flink imports. To address this issue, we are adjusting the logic: if a user employs stream processing imports like Flink with group commit mode enabled, we will automatically disable group commit mode, switching to the standard import mode instead. This is the essence of this PR.
2024-02-26 19:07:10 +08:00
8f77e6363a [Feature](function) Support xxhash function like murmur hash function (#31193) 2024-02-23 19:03:28 +08:00
90ab5ec2d9 [fix](invert index) fix the error issue in the unit test remove_element_only_in_table (#31238) 2024-02-22 13:01:49 +08:00
ad07dec0ed [Improve](InPredict) enhance in predict with struct type (#30840) 2024-02-22 13:01:49 +08:00
b66583551c [fix](group_commit)Fix bound checking problem when reading wal block (#31112) 2024-02-22 13:01:48 +08:00
f2a38e6345 [chore](columns) remove update_hashes_with_value for SipHash (#31224) 2024-02-22 13:01:48 +08:00
1abe9d4384 [fix](memory) Fix LRU cache stale sweep (#31122)
Remove LRUCacheValueBase, put last_visit_time into LRUHandle, and automatically update timestamp to last_visit_time during cache insert and lookup.

Do not rely on external modification of last_visit_time, which is often forgotten.
2024-02-21 17:01:29 +08:00
a8d8c6a271 [fix](file-writer) opt s3 file writer and fix empty file related issue #28983 #30703 #31169 (#31213)
* (feature)(cloud) Use dynamic allocator instead of static buffer pool for better elasticity. (#28983)

* [fix](outfile) Fix unable to export empty data (#30703)

Issue Number: close #30600
Fix unable to export empty data to hdfs / S3, this behavior is inconsistent with version 1.2.7,
version 1.2.7 can export empty data to hdfs/ S3, and there will be exported files on S3/HDFS.

* [fix](file-writer) avoid empty file for segment writer (#31169)

---------

Co-authored-by: AlexYue <yj976240184@gmail.com>
Co-authored-by: zxealous <zhouchangyue@baidu.com>
2024-02-21 16:48:54 +08:00
7a1bd6abb0 [improvment](group_commit) Refector scan wal function (#30939)
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
2024-02-20 09:12:38 +08:00
Pxl
bb4575a392 [Improvement](join) optimization for build_side_output_column (#30826)
optimization for build_side_output_column
2024-02-19 17:22:03 +08:00
6cf7468073 [enhancement](function) change some function nullable mode (#30991)
change some function nullable mode
2024-02-18 14:45:25 +08:00
68102fd531 [Fix](auto-partition) fix a concurrent bug of extremely long values (#31005) 2024-02-18 14:45:25 +08:00
b5012dc55a [Enhancement](group commit) optimize pre allocated calculation (#30893) 2024-02-18 11:50:17 +08:00
45b4189bb6 [Refactor](opt) Opt rf and remove unless code (#30900)
Opt rf and remove unless code
2024-02-18 11:50:16 +08:00
8ff8d94697 [fix](ip) change IPv6 to little-endian byte order storage (like IPv4) (#30730) 2024-02-05 21:56:57 +08:00
Pxl
0f47f7f389 [Feature](runtime filter) normalize ignore runtime filter (#30152)
normalize ignore runtime filter
2024-02-03 20:24:39 +08:00
3315c16383 [enhance](function) refactor from_format_str and support more format (#30452) 2024-02-01 19:08:37 +08:00
e9c112b843 [Refact](inverted index) refact inverted index cache to decouple with reader (#30574) 2024-02-01 19:00:50 +08:00
49d17f2be2 [fix](move-memtable) fix potential duplicate of TabletStream profile (#30397) 2024-01-30 15:30:14 +08:00
ccde65b942 [fix](Cooldown) Enhance calculate logic of _has_data_to_cooldown (#30244) (#30299) 2024-01-25 13:25:34 +08:00
2e6a00690f [Fix](smooth-upgrade) fix unnecessary high version of smooth upgrade (#30283)
fix unnecessary high version of smooth upgrade
2024-01-25 13:24:09 +08:00
1a51d04cb8 [fix](move-memtable) fix schema use-after-free in delta writer v2 (#30254) 2024-01-24 10:00:25 +08:00
Pxl
02a27a587a remove some unused member function of IFunctionBase (#30260) 2024-01-24 09:59:45 +08:00
f7a340a2df [improve](move-memtable) add cancel method to load stream stub (#29994) 2024-01-16 20:23:09 +08:00
e7b221ba66 [fix](be-ut) Fix unstable test cases (#29896)
The following cases are unstable.

1. LoadStreamMgrTest
2. TaskWorkerPoolTest.PriorTaskWorkerPool

Rationales

1. LoadStreamMgrTest
It is related to timeout. If we investigate the examples in BRPC, we will find the timeout is usually set to 0 rather than a specific number.
2. TaskWorkerPoolTest.PriorTaskWorkerPool
The order of the threads for the lock contentions is undetermined.
2024-01-16 18:30:23 +08:00
d494674ff4 [opt](parquet-reader) Opt parquet decimal type reading. (#29825) 2024-01-12 13:58:19 +08:00
d525f576e1 [improve] Use lru cache to count the number of column in tablet schema to control memory (#29668) 2024-01-12 13:58:19 +08:00
0d691c638b [Feature](profile)Support report runtime workload statistics #29591 2024-01-12 11:59:27 +08:00
81680383e6 [UT](wal) Add wal dirs info be ut (#29759) 2024-01-12 11:57:16 +08:00
0d16ec7345 [improvement](cooldown) do not cooldown tablet without cold data (#29690) 2024-01-12 11:57:16 +08:00
7c7dbf15bc [feature](merge-cloud) Decouple Tablet/TabletManager/TxnManager from global StorageEngine instance (#29736) 2024-01-12 11:57:16 +08:00
fc4ca712ed [bugfix](core) using weak ptr in data stream receiver to avoid runtime state is deconstructed (#29410) 2024-01-12 11:48:39 +08:00
7287c0ca15 [Opt](exec)(multi-catalog) Opt date type reading. (#29571) 2024-01-12 11:48:39 +08:00
48f58510a8 [refactor](tabletwriter) make tablet writer's rpc callback safe, could exit any time (#29684)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-01-12 11:46:29 +08:00
28dca66c06 Fix BE UT macOS compile (#29663) 2024-01-12 11:44:20 +08:00
c9e2f1934a [opt](file cache) make the cache queue percentage configurable (#29537) 2024-01-12 11:36:58 +08:00
c58d18147b [refactor](resultwriter) rename append_block to write method is more meaningful (#29635)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-01-08 00:00:33 +08:00
0b731800a0 [enhancement](group_commit) refector wal manager code (#29560) 2024-01-07 18:54:41 +08:00
b0cac0014d [enhance](FS) Improve FS error code (#29432) 2024-01-06 21:17:22 +08:00
85dd606fd1 [fix](group_commit) Fix group_commit ut (#29587) 2024-01-06 18:11:13 +08:00
a0c3ddf902 [fix](memory) Fix LRUCacheType::NUMBER charge (#29588)
if LRUCacheType::NUMBER, charge not add handle_size, because charge at this time is no longer the memory size, but an independent weight.
2024-01-06 10:37:56 +08:00
f40cce1406 [Fix](partition) Skip rowset partition id eq 0 smaller than config wh… (#29510) 2024-01-05 19:39:51 +08:00
e3c9f535dc [refactor](wal) refactor some wal code (#29434) 2024-01-03 14:45:57 +08:00
69a01e0cf5 [improve](move-memtable) skip load stream stub close wait when cancel (#29427) 2024-01-02 23:35:50 +08:00
243b075eef [improve](move-memtable) report failed tablets with status (#29351) 2024-01-02 20:24:53 +08:00
706463781c [refactor](group commit) refactor group commit wal code (#29375) 2024-01-02 15:52:03 +08:00
b7487430da Revert "[improve](move-memtable) cancel load rapidly when stream close wait (#29322)" (#29371)
This reverts commit bbf58c5aa42d40e66bc6ccc9ed91a4fcb4bdfff7.
2024-01-02 11:32:14 +08:00
bbf58c5aa4 [improve](move-memtable) cancel load rapidly when stream close wait (#29322) 2023-12-31 16:26:41 +08:00
7623b5cc31 [cleanup](move-memtable) remove namespace stream_load (#27441) 2023-12-30 20:08:23 +08:00