Commit Graph

371 Commits

Author SHA1 Message Date
4d980b8235 [feature](http action)Add http action to show nested inverted index file (#38272) (#38672)
backport #38272
2024-08-01 19:30:59 +08:00
d8744cd3d0 [Opt](load) don't print stack when some errors occur for stream load #38332 (#38418)
cherry pick from #38332
2024-07-28 19:04:24 +08:00
b7c038eb11 [fix]fix be core when migration tablet to other disk (#37712) (#38062)
![image](https://github.com/user-attachments/assets/01f67160-3ebe-41a1-ac79-a7173d14605c)
The asynchronous task reference captures a local variable
Issue Number: close #36809

Co-authored-by: z404289981 <123865460+z404289981@users.noreply.github.com>
Co-authored-by: huronghui <huronghui@huronghuideMac-mini.local>
2024-07-18 17:51:32 +08:00
e5339a4014 [feature](ES Catalog)Support control scroll level by config #37180 (#37290)
## Proposed changes

backport #37180
2024-07-15 16:41:38 +08:00
326b40cde2 [branch-2.1](memory) Add HTTP API to clear data cache (#37704)
pick #36599

Co-authored-by: Gabriel <gabrielleebuaa@gmail.com>
2024-07-12 17:21:52 +08:00
a61030215e [branch-2.1](memory) Support make all memory snapshots (#37705)
pick #36679
2024-07-12 16:21:37 +08:00
ef031c5fb2 [branch-2.1](memory) Fix reserve memory compatible with memory GC and logging (#37682)
pick
#36307
#36412
2024-07-12 11:43:26 +08:00
5280e277e7 [chore](be) Acquire and check MD5 digest of the file to download (#37418)
Cherry-pick #35807, #36621, #36726
2024-07-08 18:55:35 +08:00
612f2ae961 [feature](api) add BE HTTP /api/load_streams (#36312) (#36338)
cherry-pick #36312
2024-06-16 22:09:04 +08:00
238e218312 [fix](httpapi) restore compaction/run_status api can show be's overall compaction status and refactor code (#35409) 2024-05-28 09:43:43 +08:00
6e17dc1e87 (cherry-pick)[branch-2.1] add calc tablet file crc and fix single compaction test #33076 #34915 (#35215)
* [fix](compaction test) show single replica compaction status and fix test (#33076)
* [improve](http action) add http interface to calculate the crc of all files in tablet (#34915)
2024-05-26 17:15:09 +08:00
95593f8e19 [config](move-memtable) support HTTP header load_stream_per_node in stream load (#34444) 2024-05-10 22:05:35 +08:00
c22f42121b [fix](compaction test) show single replica compaction status and fix test (#33076) (#34285) (#34438) 2024-05-06 21:00:34 +08:00
b15fc2a906 [Cherry-pick](branch-2.1) Pick #34043 and #34112 (#34318)
* [Enhancement](full compaction) Add run status support for full compaction (#34043)

* The usage is `curl http://{ip}:{host}/api/compaction/run_status?tablet_id={tablet_id}`
e.g. `curl http://127.0.0.1:8040/api/compaction/run_status?tablet_id=10084`

If full compaction is running, the output will be
```
{
"status" : "Success",
"run_status" : true,
"msg" : "compaction task for this tablet is running",
"tablet_id" : 10084,
"compact_type" : "full"
}
```
else the ouput will be
```
{
"status" : "Success",
"run_status" : false,
"msg" : "compaction task for this tablet is not running",
"tablet_id" : 10084,
"compact_type" : "full"
}
```

* 2

* 2

* [Fix](partial update) Fix rowset not found error when doing partial update (#34112)

Cause: In the logic of partial column updates, the existing data columns are read first, and then the data is supplemented and written back. During the reading process, initialization involves initially fetching rowset IDs, and the actual rowset object is fetched only when needed later. However, between fetching the rowset IDs and the rowset object, compaction may occur, turning the old rowset into a stale rowset. If too much time passes, the stale rowset might be directly deleted. Thus, when the rowset object is needed for an update, it cannot be found. Although the update operation with partial column logic should be able to read all keys and should not encounter new keys, if the rowset disappears, the Backend (BE) will consider these keys as missing. Consequently, it will check whether other columns have default values or are nullable. If this check fails, the aforementioned error is thrown.

Solution: To avoid such issues during partial column updates, the initialization step should involve fetching both the rowset IDs and the shared pointer to the rowset object simultaneously. This ensures that the rowset can always be found during data retrieval.
2024-04-30 07:26:23 +08:00
1ca96a1611 [fix](stream-load) fix stream load and http stream metric error #33899 2024-04-20 11:43:49 +08:00
48880c3e1a [Fix](timezone) fix miss of expected rounding of Date type with timezone #33553 2024-04-17 23:42:11 +08:00
f3a60374b5 [branch-2.1](memory) Fix EngineStorageMigrationTask MemTracker attach thread (#33509) 2024-04-11 08:35:13 +08:00
90ee9c171e [bug](mem_tracker) fix mem_tracker dcheck failed as not used correctly (#33349) (#33486)
* [bug](mem_tracker) fix mem_tracker dcheck failed as not used correctly
2024-04-10 22:22:12 +08:00
Pxl
8fd6d4c41b [Chore](build) add -Wconversion and remove some unused code (#33127)
add -Wconversion and remove some unused code
2024-04-10 15:26:08 +08:00
cf7595d423 [opt](memory) Optimize mem tracker accuracy (#32039) (#33140) 2024-04-10 11:42:19 +08:00
ab512f935c [pipelineX](api) Add api for long-running tasks (#32459) 2024-03-21 14:07:24 +08:00
0635a8716c [improve](group commit) Group commit support chunked stream load in flink (#32135) 2024-03-21 14:07:24 +08:00
xy
e6f9bd4023 [Opt](exec) Reduce unnecessary find function calls (#32165) 2024-03-15 18:00:57 +08:00
5f125bbaaa [improvement](binlog)Support inverted index in CCR (#31743) (#32101) 2024-03-12 15:34:08 +08:00
da5a40077f [fix](http stream) http stream support memtable_on_sink_node header (#31866) 2024-03-07 16:53:19 +08:00
d8b9909675 [Fix](Status) Handle returned Status correctly #31434 2024-03-01 04:25:43 +08:00
b177b26d39 [branch-2.1](tracing) Pick pipeline tracing and relative bugfix (#31367)
* [Feature](pipeline) Trace pipeline scheduling (part I) (#31027)

* [fix](compile) Fix performance compile fail #31305

* [fix](compile) Fix macOS compilation issues for PURE macro and CPU core identification (#31357)

* [fix](compile) Correct PURE macro definition to fix compilation on macOS

* 2

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2024-02-29 08:42:35 +08:00
747faeed17 [Enhancement](group commit) optimize some group commit code (#31392)
This PR optimizes some of the logic related to group commit:
1. Improved the error handling when there is insufficient WAL space during import.
2. Accounted for cases where the content length is negative during import.
3. Added missing error log printing in `group_commit_mgr.cpp`.
2024-02-28 13:05:57 +08:00
48804a978a [Fix](group commit) Fix group commit flink error message (#31350)
* When using stream processing frameworks like Flink with group commit mode enabled, the uncertain size of imported data makes such behavior prohibitive. Previously, to simplify the process, the error message for excessive data volume during streamload was combined with the one for group commit mode, leading to confusion for users when encountering errors indicating the data volume is too large during Flink imports. To address this issue, we are adjusting the logic: if a user employs stream processing imports like Flink with group commit mode enabled, we will automatically disable group commit mode, switching to the standard import mode instead. This is the essence of this PR.
2024-02-26 19:07:10 +08:00
822f2b1255 [improve](stream-load) add observability on receiving HTTP request #30432 2024-02-01 19:01:08 +08:00
2a81ab70e4 [Enhancement](group commit) Optimize WAL insufficient space error messages (#30614) 2024-02-01 19:00:50 +08:00
d61fec3ff8 (feature)(LOG) Support to change be vlog level dynamically using http (#28785) 2024-01-06 20:12:35 +08:00
3888a7cc0b [fix](group_commit) Fix check auth error when relaying wal (#29461) 2024-01-03 23:19:16 +08:00
e3c9f535dc [refactor](wal) refactor some wal code (#29434) 2024-01-03 14:45:57 +08:00
be1d9c3358 [fix](memory) Fix mem tracker web page notice #29361 2024-01-03 11:25:00 +08:00
b3f69211b7 [refactor](wal) move group commit load content length to runtime state (#29188) 2024-01-02 15:53:38 +08:00
706463781c [refactor](group commit) refactor group commit wal code (#29375) 2024-01-02 15:52:03 +08:00
018d8b1326 [fix](doc) spell errors fixes and align with code log for check-rpc-channel. (#28151) 2023-12-30 20:56:56 +08:00
03901b9a7a [enhancement](group_commit): refector relay wal code (#29183) 2023-12-30 12:59:46 +08:00
e7d67e9411 [fix](be) resolves some unused-raii and used-after-moved issues (#29285) 2023-12-30 12:14:49 +08:00
9be0f04506 (improv)[group commit] refactor some group commit code (#29180) 2023-12-29 00:26:10 +08:00
9ff8bd2e9c [Enhancement](Wal)Support dynamic wal space limit (#27726) 2023-12-27 11:51:32 +08:00
f374beaa4e [fix](log) regularise some BE error type and fix a load task check #28729 2023-12-25 10:45:19 +08:00
xy
e59c707022 [fix](stream_load)fix bug for stream (#27752)
1. forbid thed  stream_load without content-length or chunked Transfer Encoding
2. forbid thed  stream_load both with content-length and chunked Transfer Encoding

Co-authored-by: xingying01 <xingying01@corp.netease.com>
2023-12-20 09:47:52 +08:00
f9ddf8c7ef [improvement](be report) add be report http (#28424) 2023-12-19 10:39:19 +08:00
ee24667b9f [fix](group commit) Fix some group commit problems (#28319) 2023-12-14 14:38:56 +08:00
d4f89389e3 [improve](group commit) Group commit support skip wal (#27957) 2023-12-11 19:38:32 +08:00
605257ccb7 [Enhancement](group commit) Add regression case for wal limit (#27949) 2023-12-06 14:23:50 +08:00
1afdbfe723 [enhance](BE) Refactor TaskWorkerPool (#27555) 2023-12-04 21:46:10 +08:00
a64656748b [Enhancenment](wal) disable group commit when streamload size is too large (#27781) 2023-12-03 23:05:11 +08:00