894bae4ebf
[improvement](publish version) publish txn fail retry do not wait ( #28441 )
2023-12-16 22:31:10 +08:00
82a91380e6
[enhancement](compaction) Add support for limiting low priority compaction scheduling ( #27648 )
2023-12-14 18:31:23 +08:00
e6e8632167
[improvement](merge-on-write) Optimize publish when there are missing versions ( #28012 )
...
1. Do not retry publishing on be When there are too many missing versions, just
add to async publish task.
2. To reduce memory consumption, clean up the tasks when there are too many
async publish tasks.
2023-12-13 16:59:25 +08:00
c4e484916b
[Fix](table property) Fix table property disable_auto_compaction ( #27853 )
2023-12-11 20:48:11 +08:00
8526b9ffbe
[imporvement](table property) support for alter table property disable_auto_compaction ( #27961 )
...
in some case, some tablets may cause coredump or OOM when compaction, and it is necessary to manually close the compaction of a specific table by 'disable_auto_compaction' to make be service available
This commit allow modify disable_auto_compaction table property in schema change.
---------
Signed-off-by: nextdreamblue <zxw520blue1@163.com >
2023-12-07 15:08:39 +08:00
3595f21405
[improvement](executor)clear unused cgroup path ( #27798 )
...
* clear unused cgroup path
* use C++ api
* add gcc header
2023-12-05 14:18:23 +08:00
d69cdf8635
[improve](heartbeat) show more info when receive invalid cluster id ( #27975 )
2023-12-05 11:10:22 +08:00
1afdbfe723
[enhance](BE) Refactor TaskWorkerPool ( #27555 )
2023-12-04 21:46:10 +08:00
0b7becd4b7
[fix](executor)Fix memtracker not set to task group #27699
2023-11-30 22:35:51 +08:00
f565f60bc3
[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default ( #27587 )
2023-11-28 13:02:30 +08:00
baadc14e60
[Enhancement](function) support unix_timestamp with float ( #26827 )
...
---------
Co-authored-by: YangWithU <plzw8@outlook.com >
2023-11-27 09:58:53 +08:00
be7273da83
[refactor](executor)Refactor workload meta update to be #26710
2023-11-18 11:19:38 +08:00
7754791146
[improvement](disk balance) Prevent duplicate disk balance tasks afte… ( #25990 )
2023-11-10 10:14:42 +08:00
6637f9c15f
Add enable_cgroup_cpu_soft_limit ( #26510 )
2023-11-08 15:52:13 +08:00
0a4863e859
fix mac compile ( #25724 )
2023-10-30 18:50:53 +08:00
1ba8a9bae4
[feature-wip](executor)Fe send topic info to be ( #25798 )
2023-10-26 15:52:48 +08:00
d6c64d305f
[chore](log) Add log to trace query execution #25739
2023-10-26 14:09:25 +08:00
2e2d5bcba2
[Improvements](status) catch some error status ( #25677 )
...
catch some error status
2023-10-23 10:19:08 +08:00
fbc448520a
[feature](ColdHeatSeperation) Support to upload cold data to HDFS ( #22048 )
2023-10-22 21:04:43 +08:00
54780c62e0
[improvement](executor)Using cgroup to implement cpu hard limit ( #25489 )
...
* Using cgroup to implement cpu hard limit
* code style
2023-10-19 18:56:26 +08:00
9c9fc84f39
[feature](merge-cloud) Abstract BaseTablet for CloudTablet ( #24929 )
2023-10-18 20:29:04 +08:00
f75ee49cb4
[chore](fmt) Remove stringstream by fmt ( #25474 )
...
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com >
2023-10-16 21:31:54 +08:00
f9df3bae61
[Enhancement](functions) change some nullable mode and clear some smooth upgrade ( #25334 )
2023-10-16 19:50:17 +08:00
6757d2f361
Revert "[Enhancement](show-backends-disks) Add show backends disks ( #24229 )" ( #25389 )
...
This reverts commit 21223e65c59c23cfcb9e8ab610ea321168bcb75a.
2023-10-13 14:08:45 +08:00
21223e65c5
[Enhancement](show-backends-disks) Add show backends disks ( #24229 )
...
* Add statement to query disk information corresponding to data directory of BE node
[msyql]->'show backends disks;'
+-----------+-------------+------------------------------+---------+----------+---------------+-------------+-------------------+---------+
| BackendId | Host | RootPath | DirType | DiskState| TotalCapacity | UsedCapacity| AvailableCapacity | UsedPct |
+-----------+-------------+------------------------------+---------+----------+---------------+-------------+-------------------+---------+
| 10002 | 10.xx.xx.90 | /home/work/output/be/storage | STORAGE | ONLINE | 7.049 TB | 2.478 TB | 4.571 TB | 35.16 % |
| 10002 | 10.xx.xx.90 | /home/work/output/be | DEPLOY | ONLINE | 7.049 TB | 2.478 TB | 4.571 TB | 35.16 % |
| 10002 | 10.xx.xx.90 | /home/work/output/be/log | LOG | ONLINE | 7.049 TB | 2.478 TB | 4.571 TB | 35.16 % |
+-----------+-------------+------------------------------+---------+----------+---------------+-------------+-------------------+---------+
2023-10-12 20:24:45 +08:00
642e5cdb69
[Fix](Status) Make Status [[nodiscard]] and handle returned Status correctly ( #23395 )
2023-09-29 22:38:52 +08:00
188d9ab94e
[enhancement](statistics) collect table level loaded rows on BE to make RPC light weight ( #24609 )
2023-09-28 10:51:50 +08:00
c9b2f4cb92
[workload](pipeline) Add cgroup cpu controller ( #24052 )
2023-09-21 21:49:33 +08:00
fc12362a6d
[feature-wip](arrow-flight)(step2) FE support Arrow Flight server ( #24314 )
...
This is a POC, the design documentation will be updated soon
2023-09-20 14:42:54 +08:00
b092bdaabf
[feature](load) collect loaded rows on table level after txn published ( #24346 )
...
As title.
Stream load 20 lines
```
2023-09-14 11:40:04,186 DEBUG (PUBLISH_VERSION|23) [DatabaseTransactionMgr.updateCatalogAfterVisible():1769] table id to loaded rows:{51016=20}
```
```
mysql> select count(*) from dup_tbl_basic;
+----------+
| count(*) |
+----------+
| 20 |
+----------+
1 row in set (0.05 sec)
```
2023-09-19 12:00:08 +08:00
d20365cdcf
[fix](transaction) fix publish txn fake succ ( #24273 )
2023-09-14 21:04:59 +08:00
32dd088d30
[fix](inverted)fix inverted index report be hostname error #23872
...
when inverted index report task status to fe,not put the correct host
delete unused _backend
2023-09-05 10:52:23 +08:00
1d1a9e2bfc
[improvement](graceful shutdown) waiting for all query finished when graceful shutdown ( #23865 )
...
In some cloud native deployment scenario, BE(especially the Compute Node BE) will be add to cluster and remove from cluster very frequently. User's query will fail if there is a fragment is running on the shutting down BE. Users could use stop_be.sh --grace, then BE will wait all running queries to stop to avoiding running query failure, but if the waiting time exceed the limit, then be will exit directly. During this period, FE will not send any queries to BE and waiting for all running queries to stop
2023-09-05 09:52:28 +08:00
acbd8ca185
[improvement](show backends) show backends print trash used ( #23792 )
2023-09-03 20:30:58 +08:00
25b6e4deb2
[fix](daemon) Fix incorrect initialization order of daemon services ( #23578 )
...
Current initialization dependency:
Daemon ───┬──► StorageEngine ──► ExecEnv ──► Disk/Mem/CpuInfo
│
│
BackendService ─┘
However, original code incorrectly initialize Daemon before StorageEngine.
This PR also stop and join threads of daemon services in their dtor, to ensure Daemon services release resources in reverse order of initialization via RAII.
2023-08-31 19:46:38 +08:00
c083336bbe
[Improvement](pipeline) Cancel outdated query if original fe restarts ( #23582 )
...
If any FE restarts, queries that is emitted from this FE will be cancelled.
Implementation of #23704
2023-08-31 17:58:52 +08:00
f35ab37e1e
[Bug](materialized-view) fix load db use analyzer to analyze diffrent metaindex ( #23673 )
...
fix load db use analyzer to analyze diffrent metaindex
2023-08-31 12:35:38 +08:00
da9eb79ac4
[Enhancement](Schema hash) Remove schema hash in tablet info ( #23516 )
2023-08-29 10:05:12 +08:00
153e8f0f72
[imporvement](table property) support for alter table property: skip wirte index , single compaction ( #23475 )
2023-08-26 23:52:09 +08:00
17e7c1ca53
[fix](fqdn)Fqdn with ipv6 ( #22454 )
...
now,`hostname_to_ip` only can resolve `ipv4`,Therefore, a method is provided to parse ipv4 or ipv6 based on parameters。
when `_heartbeat` call `hostname_to_ip`,Resolve to ipv4 or ipv6, determined by `BackendOptions.is_bind_ipv6` Decision
Additionally, a method is provided to first attempt to parse the host into ipv4, and then try ipv6 if it fails
2023-08-25 21:24:55 +08:00
0967d7ec04
[improvement](agg) Do not serialize bitmap to string ( #23172 )
2023-08-21 10:10:15 +08:00
b9b9071c9b
[improvement](create partition) create partition require quorum replicas succ ( #22554 )
2023-08-11 11:59:05 +08:00
ec187662be
use correct bool value ( #22507 )
2023-08-03 20:09:57 +08:00
19d1f49fbe
[improvement](compaction) compaction policy and options in the properties of a table ( #22461 )
2023-08-01 22:02:23 +08:00
06e4061b94
[enhance](ColdHeatSeparation) carry use path style info along with cold heat separation to support using minio ( #22249 )
2023-07-30 21:03:33 +08:00
c31e826756
[opt](config) rename alter_inverted_index_worker_count to alter_index_worker_count, and add docs ( #21985 )
2023-07-20 17:50:04 +08:00
ca71048f7f
[Chore](status) avoid empty error msg on status ( #21454 )
...
avoid empty error msg on status
2023-07-11 13:48:16 +08:00
f8a2c66174
[refactor](planner) refactor automatically set instance_num ( #21640 )
...
refactor automatically set instance_num
2023-07-08 21:59:17 +08:00
181dad4181
[fix](executor) make elt / repeat smooth upgrade. ( #21493 )
...
BE : 2.0，FE : 1.2
before
mysql [(none)]>select elt(1, 'aaa', 'bbb');
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Function elt get failed, expr is VectorizedFnCall[elt](arguments=,return=String) and return type is String.
mysql [test]> INSERT INTO tbb VALUES (1, repeat("test1111", 8192))(2, repeat("test1111", 131072));
mysql [test]>select k1, md5(v1), length(v1) from tbb;
+------+----------------------------------+--------------+
| k1 | md5(`v1`) | length(`v1`) |
+------+----------------------------------+--------------+
| 1 | d41d8cd98f00b204e9800998ecf8427e | 0 |
| 2 | d41d8cd98f00b204e9800998ecf8427e | 0 |
+------+----------------------------------+--------------+
now
mysql [test]>select elt(1, 'aaa', 'bbb');
+----------------------+
| elt(1, 'aaa', 'bbb') |
+----------------------+
| aaa |
+----------------------+
mysql [test]>select k1, md5(v1), length(v1) from tbb;
+------+----------------------------------+--------------+
| k1 | md5(`v1`) | length(`v1`) |
+------+----------------------------------+--------------+
| 1 | 1f44fb91f47cab16f711973af06294a0 | 65536 |
| 2 | 3c514d3b89e26e2f983b7bd4cbb82055 | 1048576 |
+------+----------------------------------+--------------+
2023-07-06 19:15:06 +08:00
1fe04b7242
[Chore](metrics) remove trace metrics code using runtime profile instead ( #21394 )
...
* commit
* fix
* format
2023-07-01 12:18:23 +08:00
