f374beaa4e
[fix](log) regularise some BE error type and fix a load task check #28729
2023-12-25 10:45:19 +08:00
894bae4ebf
[improvement](publish version) publish txn fail retry do not wait ( #28441 )
2023-12-16 22:31:10 +08:00
82a91380e6
[enhancement](compaction) Add support for limiting low priority compaction scheduling ( #27648 )
2023-12-14 18:31:23 +08:00
e6e8632167
[improvement](merge-on-write) Optimize publish when there are missing versions ( #28012 )
...
1. Do not retry publishing on be When there are too many missing versions, just
add to async publish task.
2. To reduce memory consumption, clean up the tasks when there are too many
async publish tasks.
2023-12-13 16:59:25 +08:00
c4e484916b
[Fix](table property) Fix table property disable_auto_compaction ( #27853 )
2023-12-11 20:48:11 +08:00
8526b9ffbe
[imporvement](table property) support for alter table property disable_auto_compaction ( #27961 )
...
in some case, some tablets may cause coredump or OOM when compaction, and it is necessary to manually close the compaction of a specific table by 'disable_auto_compaction' to make be service available
This commit allow modify disable_auto_compaction table property in schema change.
---------
Signed-off-by: nextdreamblue <zxw520blue1@163.com >
2023-12-07 15:08:39 +08:00
1afdbfe723
[enhance](BE) Refactor TaskWorkerPool ( #27555 )
2023-12-04 21:46:10 +08:00
7754791146
[improvement](disk balance) Prevent duplicate disk balance tasks afte… ( #25990 )
2023-11-10 10:14:42 +08:00
d6c64d305f
[chore](log) Add log to trace query execution #25739
2023-10-26 14:09:25 +08:00
fbc448520a
[feature](ColdHeatSeperation) Support to upload cold data to HDFS ( #22048 )
2023-10-22 21:04:43 +08:00
9c9fc84f39
[feature](merge-cloud) Abstract BaseTablet for CloudTablet ( #24929 )
2023-10-18 20:29:04 +08:00
6757d2f361
Revert "[Enhancement](show-backends-disks) Add show backends disks ( #24229 )" ( #25389 )
...
This reverts commit 21223e65c59c23cfcb9e8ab610ea321168bcb75a.
2023-10-13 14:08:45 +08:00
21223e65c5
[Enhancement](show-backends-disks) Add show backends disks ( #24229 )
...
* Add statement to query disk information corresponding to data directory of BE node
[msyql]->'show backends disks;'
+-----------+-------------+------------------------------+---------+----------+---------------+-------------+-------------------+---------+
| BackendId | Host | RootPath | DirType | DiskState| TotalCapacity | UsedCapacity| AvailableCapacity | UsedPct |
+-----------+-------------+------------------------------+---------+----------+---------------+-------------+-------------------+---------+
| 10002 | 10.xx.xx.90 | /home/work/output/be/storage | STORAGE | ONLINE | 7.049 TB | 2.478 TB | 4.571 TB | 35.16 % |
| 10002 | 10.xx.xx.90 | /home/work/output/be | DEPLOY | ONLINE | 7.049 TB | 2.478 TB | 4.571 TB | 35.16 % |
| 10002 | 10.xx.xx.90 | /home/work/output/be/log | LOG | ONLINE | 7.049 TB | 2.478 TB | 4.571 TB | 35.16 % |
+-----------+-------------+------------------------------+---------+----------+---------------+-------------+-------------------+---------+
2023-10-12 20:24:45 +08:00
642e5cdb69
[Fix](Status) Make Status [[nodiscard]] and handle returned Status correctly ( #23395 )
2023-09-29 22:38:52 +08:00
188d9ab94e
[enhancement](statistics) collect table level loaded rows on BE to make RPC light weight ( #24609 )
2023-09-28 10:51:50 +08:00
b092bdaabf
[feature](load) collect loaded rows on table level after txn published ( #24346 )
...
As title.
Stream load 20 lines
```
2023-09-14 11:40:04,186 DEBUG (PUBLISH_VERSION|23) [DatabaseTransactionMgr.updateCatalogAfterVisible():1769] table id to loaded rows:{51016=20}
```
```
mysql> select count(*) from dup_tbl_basic;
+----------+
| count(*) |
+----------+
| 20 |
+----------+
1 row in set (0.05 sec)
```
2023-09-19 12:00:08 +08:00
d20365cdcf
[fix](transaction) fix publish txn fake succ ( #24273 )
2023-09-14 21:04:59 +08:00
32dd088d30
[fix](inverted)fix inverted index report be hostname error #23872
...
when inverted index report task status to fe,not put the correct host
delete unused _backend
2023-09-05 10:52:23 +08:00
acbd8ca185
[improvement](show backends) show backends print trash used ( #23792 )
2023-09-03 20:30:58 +08:00
25b6e4deb2
[fix](daemon) Fix incorrect initialization order of daemon services ( #23578 )
...
Current initialization dependency:
Daemon ───┬──► StorageEngine ──► ExecEnv ──► Disk/Mem/CpuInfo
│
│
BackendService ─┘
However, original code incorrectly initialize Daemon before StorageEngine.
This PR also stop and join threads of daemon services in their dtor, to ensure Daemon services release resources in reverse order of initialization via RAII.
2023-08-31 19:46:38 +08:00
da9eb79ac4
[Enhancement](Schema hash) Remove schema hash in tablet info ( #23516 )
2023-08-29 10:05:12 +08:00
153e8f0f72
[imporvement](table property) support for alter table property: skip wirte index , single compaction ( #23475 )
2023-08-26 23:52:09 +08:00
b9b9071c9b
[improvement](create partition) create partition require quorum replicas succ ( #22554 )
2023-08-11 11:59:05 +08:00
ec187662be
use correct bool value ( #22507 )
2023-08-03 20:09:57 +08:00
19d1f49fbe
[improvement](compaction) compaction policy and options in the properties of a table ( #22461 )
2023-08-01 22:02:23 +08:00
06e4061b94
[enhance](ColdHeatSeparation) carry use path style info along with cold heat separation to support using minio ( #22249 )
2023-07-30 21:03:33 +08:00
c31e826756
[opt](config) rename alter_inverted_index_worker_count to alter_index_worker_count, and add docs ( #21985 )
2023-07-20 17:50:04 +08:00
ca71048f7f
[Chore](status) avoid empty error msg on status ( #21454 )
...
avoid empty error msg on status
2023-07-11 13:48:16 +08:00
f8a2c66174
[refactor](planner) refactor automatically set instance_num ( #21640 )
...
refactor automatically set instance_num
2023-07-08 21:59:17 +08:00
1fe04b7242
[Chore](metrics) remove trace metrics code using runtime profile instead ( #21394 )
...
* commit
* fix
* format
2023-07-01 12:18:23 +08:00
66005570c9
[fix](regression) fix p1 test_backup_restore fail caused by http download 401 invalid token error #21107
2023-06-26 12:56:46 +08:00
691a988c97
[enhancement](merge-on-write) add async publish task when version is discontinuous for merge on write table when clone ( #21025 )
...
version discontinuity may occur when clone. To deal with this case, add async publish task when version is discontinuous.
2023-06-22 21:50:14 +08:00
564b3533cf
[enhancement](merge-on-write) update publish/streamload/compaction co… ( #21040 )
2023-06-21 14:49:51 +08:00
87e3a79387
[enhancement](pk) add bvar latency recorder for pk ( #20942 )
2023-06-19 15:29:42 +08:00
ce9a20a375
[enhancement](merge-on-write) format logs about MoW and add more stats for publish ( #20853 )
2023-06-17 23:14:28 +08:00
9d41edd9eb
[Feature](binlog) Add binlog gc && Auth master_token ( #20854 )
...
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com >
2023-06-16 11:25:11 +08:00
f2025b9eed
[fix](memory) before compaction run, check memory exceed limit #20782
2023-06-14 14:20:48 +08:00
a68fc551f0
[bug](cooldown) Fix async_write_cooldown_meta and snapshot cooldowned version not continuous bug ( #20437 )
2023-06-08 15:35:35 +08:00
03cb69c0ee
[feature](backup-restore) Add local backup/restore not upload/download by broker ( #20492 )
2023-06-07 21:35:15 +08:00
b22e364cdb
[fix](log) publish version log is printed too frequently ( #20507 )
2023-06-06 20:34:38 +08:00
7dc7ed97eb
[Chore](build) remove some unused code and remove some wno ( #20326 )
...
remove some unused code about spinlock
remove some wno and fix warning
remove varadic macro usage
2023-06-05 10:48:07 +08:00
c45da40ed7
[refactor-WIP](TaskWorkerPool) add specific classes for ALTER_TABLE, CLONE, STORAGE_MEDIUM_MIGRATE task ( #20140 )
2023-05-28 19:27:08 +08:00
ae352997b4
[Enhancement](alter inverted index) Improve alter inverted index performance with light weight add or drop inverted index ( #19063 )
2023-05-28 11:23:07 +08:00
0434c6a738
[refactor-WIP](TaskWorkerPool) add specific classes for PUSH, PUBLIC_VERION, CLEAR_TRANSACTION tasks ( #19822 )
2023-05-27 22:47:45 +08:00
93933308e6
[Feature-WIP](CCR): Add ccr doris interface (WIP) ( #17881 )
2023-05-26 23:40:49 +08:00
da66a64e09
[fix](merge-on-write) return error st if check_pk_in_pre_segments failed ( #19736 )
2023-05-23 11:04:15 +08:00
e242d7dfcc
[refactor-WIP](TaskWorkerPool) add DropTableTaskPool for DROP_TABLE task ( #19793 )
2023-05-18 18:25:13 +08:00
6a5b590873
[refactor-WIP](TaskWorkerPool) add CreateTableTaskPool class for CREATE_TABLE task ( #19734 )
2023-05-18 11:43:09 +08:00
b29e87a382
[fix](load) donot hang publish due to compaction signal ( #19440 )
2023-05-17 16:20:55 +08:00
9cd7005dec
[fix](delete) notify all when there is no high priority task ( #19577 )
...
In somecases high priority threads are waked but normal are not. We
notify_all as a workaround.
2023-05-16 11:29:10 +08:00
