0934fbee7e
[improvement](query) prefer to chose tablet on alive disk #39467 ( #39654 )
...
cherry pick from #39467
2024-08-23 12:23:12 +08:00
bf1c7a1c15
[fix](clone) fix stale tablet report miss the new cloning replica #38695 ( #38839 )
...
cherry pick from #38695
2024-08-05 18:04:24 +08:00
327069fdbc
[branch-2.1](log) add tablet clear cache log ( #38713 )
2024-08-02 08:40:02 +08:00
73199122c2
[enhancement](compaction) Control the parallelism for urgent compacton tasks ( #37782 ) ( #38189 )
...
## Proposed changes
For some urgent compaction tasks, their submittion should take
parallelism into account.
Currently, we apply the control policy for data loading in specific.
Other source of urgent tasks are considered as eager.
2024-07-22 17:22:53 +08:00
22cb7b8fcb
[improvement](compaction) be do not compact invisible version to avoid query error -230 #28082 ( #36222 )
...
cherry pick from #28082
2024-06-27 13:45:21 +08:00
630fd06ccf
Pick "[feature](Resource) Support to specify the root path for hdfs resource #32632 " ( #35848 )
...
same as #32632
2024-06-05 08:59:20 +08:00
5c40e87667
[opt](s3) auto retry when meeting 429 error ( #35397 )
...
- Add 2 new BE config
- `s3_read_base_wait_time_ms` and `s3_read_max_wait_time_ms`
When meet s3 429 error, the "get" request will
sleep `s3_read_base_wait_time_ms (*1, *2, *3, *4)` ms get try again.
The max sleep time is s3_read_max_wait_time_ms
and the max retry time is max_s3_client_retry
- Add more metrics for s3 file reader
- `s3_file_reader_too_many_request`: counter of 429 error.
- `s3_file_reader_s3_get_request`: the QPS of s3 get request.
- `TotalGetRequest`: Get request counter in profile
- `TooManyRequestErr`: 429 error counter in profile
- `TooManyRequestSleepTime`: Sum of sleep time after 429 error in profile
- `TotalBytesRead`: Total bytes read from s3 in profile
2024-05-28 23:00:31 +08:00
2ed6a00fd1
[opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory ( #34985 ) ( #35070 )
2024-05-22 09:53:45 +08:00
7a40f2a547
[branch-2.1](resource)fix check available fail when s3 aws_token is set and reset as, sk faild on be. ( #34219 )
2024-05-09 19:06:14 +08:00
a050513c91
[Fix](clean trash) Fix clean trash use agent task ( #33912 ) ( #33972 )
...
* [Fix](clean trash) Fix clean trash use agent task (#33912 )
* add .h
2024-04-22 17:14:21 +08:00
58347aa2b7
[branch-2.1](bugfix) fix alter tablet not have memtracker
...
[branch-2.1](memory) Fix EngineAlterTabletTask attach memory tracker #33506
2024-04-11 09:29:42 +08:00
f3a60374b5
[branch-2.1](memory) Fix EngineStorageMigrationTask MemTracker attach thread ( #33509 )
2024-04-11 08:35:13 +08:00
cf7595d423
[opt](memory) Optimize mem tracker accuracy ( #32039 ) ( #33140 )
2024-04-10 11:42:19 +08:00
d60d804d9c
[fix](memory) Fix task repeat attach task DCHECK failed #32784 ( #33343 )
...
[branch-2.1](memory) Fix CCR task repeat attach task DCHECK failed3 #33366
2024-04-08 16:15:04 +08:00
e71e1b6f30
[enhancement](threadname) change thread name more simple in order to see it in top -H ( #31734 )
...
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-03-25 22:31:55 +08:00
ab467f53db
[fix](partition) Fix be tablet partition id eq 0 By report tablet ( #32179 ) ( #32667 )
2024-03-22 15:38:58 +08:00
0cde0cbf19
(invert index) modify of time series compaction policy
2024-03-22 08:16:30 +08:00
d8b9909675
[Fix](Status) Handle returned Status correctly #31434
2024-03-01 04:25:43 +08:00
41e31ee333
creat hdfs fs with it's resource id ( #31505 )
2024-02-28 11:33:34 +08:00
cc3c6d1479
[improvement](create tablet) backend create tablet round robin among … ( #30530 )
...
* [improvement](create tablet) backend create tablet round robin among … (#29818 )
* [improvement](create tablet) be choose disk tolerate with little skew (#30354 )
---------
Co-authored-by: yujun <yu.jun.reach@gmail.com >
2024-01-30 10:20:35 +08:00
8ca807578f
[fix](migrate disk) fix migrate disk lost data during publish version ( #29887 )
...
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com >
2024-01-16 18:37:06 +08:00
d6cb2d6d5c
[improvement](compaction) start 1 cumu compaction thread each disk by default ( #29430 )
2024-01-03 20:48:11 +08:00
e7d67e9411
[fix](be) resolves some unused-raii and used-after-moved issues ( #29285 )
2023-12-30 12:14:49 +08:00
59fdd5e42b
[fix](compaction) time series compaction policy -> compact consecutive empty rowsets ( #27299 )
...
Sometimes we need to merge a large number of empty versions to reduce meta-information.
2023-12-29 22:20:41 +08:00
f374beaa4e
[fix](log) regularise some BE error type and fix a load task check #28729
2023-12-25 10:45:19 +08:00
894bae4ebf
[improvement](publish version) publish txn fail retry do not wait ( #28441 )
2023-12-16 22:31:10 +08:00
82a91380e6
[enhancement](compaction) Add support for limiting low priority compaction scheduling ( #27648 )
2023-12-14 18:31:23 +08:00
e6e8632167
[improvement](merge-on-write) Optimize publish when there are missing versions ( #28012 )
...
1. Do not retry publishing on be When there are too many missing versions, just
add to async publish task.
2. To reduce memory consumption, clean up the tasks when there are too many
async publish tasks.
2023-12-13 16:59:25 +08:00
c4e484916b
[Fix](table property) Fix table property disable_auto_compaction ( #27853 )
2023-12-11 20:48:11 +08:00
8526b9ffbe
[imporvement](table property) support for alter table property disable_auto_compaction ( #27961 )
...
in some case, some tablets may cause coredump or OOM when compaction, and it is necessary to manually close the compaction of a specific table by 'disable_auto_compaction' to make be service available
This commit allow modify disable_auto_compaction table property in schema change.
---------
Signed-off-by: nextdreamblue <zxw520blue1@163.com >
2023-12-07 15:08:39 +08:00
1afdbfe723
[enhance](BE) Refactor TaskWorkerPool ( #27555 )
2023-12-04 21:46:10 +08:00
7754791146
[improvement](disk balance) Prevent duplicate disk balance tasks afte… ( #25990 )
2023-11-10 10:14:42 +08:00
d6c64d305f
[chore](log) Add log to trace query execution #25739
2023-10-26 14:09:25 +08:00
fbc448520a
[feature](ColdHeatSeperation) Support to upload cold data to HDFS ( #22048 )
2023-10-22 21:04:43 +08:00
9c9fc84f39
[feature](merge-cloud) Abstract BaseTablet for CloudTablet ( #24929 )
2023-10-18 20:29:04 +08:00
6757d2f361
Revert "[Enhancement](show-backends-disks) Add show backends disks ( #24229 )" ( #25389 )
...
This reverts commit 21223e65c59c23cfcb9e8ab610ea321168bcb75a.
2023-10-13 14:08:45 +08:00
21223e65c5
[Enhancement](show-backends-disks) Add show backends disks ( #24229 )
...
* Add statement to query disk information corresponding to data directory of BE node
[msyql]->'show backends disks;'
+-----------+-------------+------------------------------+---------+----------+---------------+-------------+-------------------+---------+
| BackendId | Host | RootPath | DirType | DiskState| TotalCapacity | UsedCapacity| AvailableCapacity | UsedPct |
+-----------+-------------+------------------------------+---------+----------+---------------+-------------+-------------------+---------+
| 10002 | 10.xx.xx.90 | /home/work/output/be/storage | STORAGE | ONLINE | 7.049 TB | 2.478 TB | 4.571 TB | 35.16 % |
| 10002 | 10.xx.xx.90 | /home/work/output/be | DEPLOY | ONLINE | 7.049 TB | 2.478 TB | 4.571 TB | 35.16 % |
| 10002 | 10.xx.xx.90 | /home/work/output/be/log | LOG | ONLINE | 7.049 TB | 2.478 TB | 4.571 TB | 35.16 % |
+-----------+-------------+------------------------------+---------+----------+---------------+-------------+-------------------+---------+
2023-10-12 20:24:45 +08:00
642e5cdb69
[Fix](Status) Make Status [[nodiscard]] and handle returned Status correctly ( #23395 )
2023-09-29 22:38:52 +08:00
188d9ab94e
[enhancement](statistics) collect table level loaded rows on BE to make RPC light weight ( #24609 )
2023-09-28 10:51:50 +08:00
b092bdaabf
[feature](load) collect loaded rows on table level after txn published ( #24346 )
...
As title.
Stream load 20 lines
```
2023-09-14 11:40:04,186 DEBUG (PUBLISH_VERSION|23) [DatabaseTransactionMgr.updateCatalogAfterVisible():1769] table id to loaded rows:{51016=20}
```
```
mysql> select count(*) from dup_tbl_basic;
+----------+
| count(*) |
+----------+
| 20 |
+----------+
1 row in set (0.05 sec)
```
2023-09-19 12:00:08 +08:00
d20365cdcf
[fix](transaction) fix publish txn fake succ ( #24273 )
2023-09-14 21:04:59 +08:00
32dd088d30
[fix](inverted)fix inverted index report be hostname error #23872
...
when inverted index report task status to fe,not put the correct host
delete unused _backend
2023-09-05 10:52:23 +08:00
acbd8ca185
[improvement](show backends) show backends print trash used ( #23792 )
2023-09-03 20:30:58 +08:00
25b6e4deb2
[fix](daemon) Fix incorrect initialization order of daemon services ( #23578 )
...
Current initialization dependency:
Daemon ───┬──► StorageEngine ──► ExecEnv ──► Disk/Mem/CpuInfo
│
│
BackendService ─┘
However, original code incorrectly initialize Daemon before StorageEngine.
This PR also stop and join threads of daemon services in their dtor, to ensure Daemon services release resources in reverse order of initialization via RAII.
2023-08-31 19:46:38 +08:00
da9eb79ac4
[Enhancement](Schema hash) Remove schema hash in tablet info ( #23516 )
2023-08-29 10:05:12 +08:00
153e8f0f72
[imporvement](table property) support for alter table property: skip wirte index , single compaction ( #23475 )
2023-08-26 23:52:09 +08:00
b9b9071c9b
[improvement](create partition) create partition require quorum replicas succ ( #22554 )
2023-08-11 11:59:05 +08:00
ec187662be
use correct bool value ( #22507 )
2023-08-03 20:09:57 +08:00
19d1f49fbe
[improvement](compaction) compaction policy and options in the properties of a table ( #22461 )
2023-08-01 22:02:23 +08:00
06e4061b94
[enhance](ColdHeatSeparation) carry use path style info along with cold heat separation to support using minio ( #22249 )
2023-07-30 21:03:33 +08:00
