ab512f935c
[pipelineX](api) Add api for long-running tasks ( #32459 )
2024-03-21 14:07:24 +08:00
0635a8716c
[improve](group commit) Group commit support chunked stream load in flink ( #32135 )
2024-03-21 14:07:24 +08:00
e6f9bd4023
[Opt](exec) Reduce unnecessary find function calls ( #32165 )
2024-03-15 18:00:57 +08:00
5f125bbaaa
[improvement](binlog)Support inverted index in CCR ( #31743 ) ( #32101 )
2024-03-12 15:34:08 +08:00
da5a40077f
[fix](http stream) http stream support memtable_on_sink_node header ( #31866 )
2024-03-07 16:53:19 +08:00
d8b9909675
[Fix](Status) Handle returned Status correctly #31434
2024-03-01 04:25:43 +08:00
b177b26d39
[branch-2.1](tracing) Pick pipeline tracing and relative bugfix ( #31367 )
...
* [Feature](pipeline) Trace pipeline scheduling (part I) (#31027 )
* [fix](compile) Fix performance compile fail #31305
* [fix](compile) Fix macOS compilation issues for PURE macro and CPU core identification (#31357 )
* [fix](compile) Correct PURE macro definition to fix compilation on macOS
* 2
---------
Co-authored-by: zy-kkk <zhongyk10@gmail.com >
2024-02-29 08:42:35 +08:00
747faeed17
[Enhancement](group commit) optimize some group commit code ( #31392 )
...
This PR optimizes some of the logic related to group commit:
1. Improved the error handling when there is insufficient WAL space during import.
2. Accounted for cases where the content length is negative during import.
3. Added missing error log printing in `group_commit_mgr.cpp`.
2024-02-28 13:05:57 +08:00
48804a978a
[Fix](group commit) Fix group commit flink error message ( #31350 )
...
* When using stream processing frameworks like Flink with group commit mode enabled, the uncertain size of imported data makes such behavior prohibitive. Previously, to simplify the process, the error message for excessive data volume during streamload was combined with the one for group commit mode, leading to confusion for users when encountering errors indicating the data volume is too large during Flink imports. To address this issue, we are adjusting the logic: if a user employs stream processing imports like Flink with group commit mode enabled, we will automatically disable group commit mode, switching to the standard import mode instead. This is the essence of this PR.
2024-02-26 19:07:10 +08:00
822f2b1255
[improve](stream-load) add observability on receiving HTTP request #30432
2024-02-01 19:01:08 +08:00
2a81ab70e4
[Enhancement](group commit) Optimize WAL insufficient space error messages ( #30614 )
2024-02-01 19:00:50 +08:00
d61fec3ff8
(feature)(LOG) Support to change be vlog level dynamically using http ( #28785 )
2024-01-06 20:12:35 +08:00
3888a7cc0b
[fix](group_commit) Fix check auth error when relaying wal ( #29461 )
2024-01-03 23:19:16 +08:00
e3c9f535dc
[refactor](wal) refactor some wal code ( #29434 )
2024-01-03 14:45:57 +08:00
be1d9c3358
[fix](memory) Fix mem tracker web page notice #29361
2024-01-03 11:25:00 +08:00
b3f69211b7
[refactor](wal) move group commit load content length to runtime state ( #29188 )
2024-01-02 15:53:38 +08:00
706463781c
[refactor](group commit) refactor group commit wal code ( #29375 )
2024-01-02 15:52:03 +08:00
018d8b1326
[fix](doc) spell errors fixes and align with code log for check-rpc-channel. ( #28151 )
2023-12-30 20:56:56 +08:00
03901b9a7a
[enhancement](group_commit): refector relay wal code ( #29183 )
2023-12-30 12:59:46 +08:00
e7d67e9411
[fix](be) resolves some unused-raii and used-after-moved issues ( #29285 )
2023-12-30 12:14:49 +08:00
9be0f04506
(improv)[group commit] refactor some group commit code ( #29180 )
2023-12-29 00:26:10 +08:00
9ff8bd2e9c
[Enhancement](Wal)Support dynamic wal space limit ( #27726 )
2023-12-27 11:51:32 +08:00
f374beaa4e
[fix](log) regularise some BE error type and fix a load task check #28729
2023-12-25 10:45:19 +08:00
e59c707022
[fix](stream_load)fix bug for stream ( #27752 )
...
1. forbid thed stream_load without content-length or chunked Transfer Encoding
2. forbid thed stream_load both with content-length and chunked Transfer Encoding
Co-authored-by: xingying01 <xingying01@corp.netease.com >
2023-12-20 09:47:52 +08:00
f9ddf8c7ef
[improvement](be report) add be report http ( #28424 )
2023-12-19 10:39:19 +08:00
ee24667b9f
[fix](group commit) Fix some group commit problems ( #28319 )
2023-12-14 14:38:56 +08:00
d4f89389e3
[improve](group commit) Group commit support skip wal ( #27957 )
2023-12-11 19:38:32 +08:00
605257ccb7
[Enhancement](group commit) Add regression case for wal limit ( #27949 )
2023-12-06 14:23:50 +08:00
1afdbfe723
[enhance](BE) Refactor TaskWorkerPool ( #27555 )
2023-12-04 21:46:10 +08:00
a64656748b
[Enhancenment](wal) disable group commit when streamload size is too large ( #27781 )
2023-12-03 23:05:11 +08:00
9daa7dc6b5
[refactor](http) disable snapshot and get_log_file api ( #27724 )
...
Disable 2 http api by default:
1. BE's `/api/snapshot`
2. FE's `/get_log_file`
2023-11-29 16:11:51 +08:00
f565f60bc3
[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default ( #27587 )
2023-11-28 13:02:30 +08:00
840f3b6439
[pipelineX](dependency) Wake by task by read dependency ( #27260 )
2023-11-20 22:00:59 +08:00
b1eef30b49
[pipelineX](dependency) Wake up task by dependencies ( #26879 )
...
---------
Co-authored-by: Mryange <2319153948@qq.com >
2023-11-18 03:20:24 +08:00
e1b180d53d
[improve](streamload) Explicitly judge the return value of close #27134
2023-11-17 14:17:09 +08:00
e1ba471727
[fix](send_batch_parallelism) add test case for send_batch_parallelism ( #26908 )
2023-11-15 14:21:58 +08:00
c0fda8c5c2
[improve](group commit) Add a swicth to wait internal group commit lo… ( #26734 )
...
* [improve](group commit) Add a swicth to make internal group commit load finish
* modify group commit tvf plan
2023-11-13 10:35:35 +08:00
d767804815
[feature](merge-cloud) Decouple rowset id generator and local rowsets gc implementation ( #25921 )
2023-11-10 10:07:02 +08:00
607a5d25f1
[feature](streamload) support HTTP request with chunked transfer ( #26520 )
2023-11-08 10:07:05 +08:00
2cc68381ec
[feature](binlog) Add ingest_binlog/http_get_snapshot limit download speed && Add async ingest_binlog ( #26323 )
2023-11-06 11:14:44 +08:00
696ecc8c83
[Chore](log) adjust error code on too many filtered rows ( #26168 )
2023-11-01 00:15:56 +08:00
88dd480c2e
[enhancement](CSV-reader) enhance err log for csv reading containing enclose or escape ( #25816 )
2023-10-24 22:10:08 +08:00
2972daaed9
[Bug](status) process error status on es_scroll_parser and compaction_action ( #25745 )
...
process error status on es_scroll_parser and compaction_action
2023-10-24 15:51:01 +08:00
2e2d5bcba2
[Improvements](status) catch some error status ( #25677 )
...
catch some error status
2023-10-23 10:19:08 +08:00
d0cd535cb9
[improvement](insert) refactor group commit stream load ( #25560 )
2023-10-20 13:27:30 +08:00
9c9fc84f39
[feature](merge-cloud) Abstract BaseTablet for CloudTablet ( #24929 )
2023-10-18 20:29:04 +08:00
cda8fb6b8b
[fix](load) return Status when error in RowsetWriter::build ( #25381 )
2023-10-17 09:40:23 +08:00
73c3e3ab55
[Feature](x-load) support config min replica num for loading data ( #21118 )
2023-10-11 21:07:35 +08:00
5a55e47acd
[Enhancement](Load) stream tvf support two phase commit ( #23800 )
2023-10-09 14:15:56 +08:00
6fe060b79e
[fix](streamload) fix http_stream retry mechanism ( #24978 )
...
If a failure occurs, doris may retry. Due to ctx->is_read_schema is a global variable that has not been reset in a timely manner, which may cause exceptions.
---------
Co-authored-by: yiguolei <676222867@qq.com >
2023-10-08 11:16:21 +08:00
