cc3c6d1479
[improvement](create tablet) backend create tablet round robin among … ( #30530 )
...
* [improvement](create tablet) backend create tablet round robin among … (#29818 )
* [improvement](create tablet) be choose disk tolerate with little skew (#30354 )
---------
Co-authored-by: yujun <yu.jun.reach@gmail.com >
2024-01-30 10:20:35 +08:00
61bab1f1c0
[Fix](inverted index) fix data size when drop inverted index ( #30327 )
2024-01-25 13:24:52 +08:00
4b4fd1a290
[improvement](log) add txn log ( #28875 )
2024-01-16 18:37:06 +08:00
8ca807578f
[fix](migrate disk) fix migrate disk lost data during publish version ( #29887 )
...
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com >
2024-01-16 18:37:06 +08:00
c599cf311d
[fix](migrate) migrate check old tablet had deleted ( #29909 )
2024-01-16 18:33:51 +08:00
e3a1138da7
[fix](migrate disk) fix tablet disk migration timeout too large ( #29895 )
2024-01-16 18:33:51 +08:00
a314491535
[Fix](inverted index) fix array inverted index builder error ( #29869 )
2024-01-12 13:58:19 +08:00
7c7dbf15bc
[feature](merge-cloud) Decouple Tablet/TabletManager/TxnManager from global StorageEngine instance ( #29736 )
2024-01-12 11:57:16 +08:00
fa4721125e
[Fix](show data) Inverted index size not added to data size after index build ( #29568 )
2024-01-05 16:21:45 +08:00
797238cbb7
[fix](merge-on-write) fix schema change may result in delete bitmap incorrect ( #29386 )
2024-01-02 23:45:04 +08:00
1aa9ac4fe4
Prevent making snapshot on remote rowset in single replica compaction ( #28716 )
2023-12-27 23:43:43 +08:00
a8e6676640
[Bug](security) BE download_files function exists log print sensitive msg #28592 ( #28594 )
2023-12-26 21:59:47 +08:00
f374beaa4e
[fix](log) regularise some BE error type and fix a load task check #28729
2023-12-25 10:45:19 +08:00
dfbf082e06
[fix](merge-on-write) migration may cause duplicate keys for mow table ( #28923 )
2023-12-23 23:37:00 +08:00
34fd376f33
[fix](publish version) fix publish fail but return ok ( #28425 )
2023-12-21 11:10:08 +08:00
b142ade69e
[refactor](renamefile) rename some files according to the class names ( #28606 )
2023-12-19 14:10:11 +08:00
e6e8632167
[improvement](merge-on-write) Optimize publish when there are missing versions ( #28012 )
...
1. Do not retry publishing on be When there are too many missing versions, just
add to async publish task.
2. To reduce memory consumption, clean up the tasks when there are too many
async publish tasks.
2023-12-13 16:59:25 +08:00
ebed055d2b
[chore](clone) rename clone request field ( #27591 )
2023-12-08 11:53:57 +08:00
1afdbfe723
[enhance](BE) Refactor TaskWorkerPool ( #27555 )
2023-12-04 21:46:10 +08:00
7398c3daf1
[Feature-Variant](Variant Type) support variant type query and index ( #27676 )
2023-11-29 10:37:28 +08:00
f565f60bc3
[refactor](standard)BE:Initialize pointer variables in the class to nullptr by default ( #27587 )
2023-11-28 13:02:30 +08:00
2f04873da9
[fix](clone) Fix engine_clone file exist ( #27361 )
...
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com >
2023-11-21 20:30:27 +08:00
e58983e59c
[improvement](publish version) reduce publish version not exist log ( #27332 )
2023-11-21 18:00:01 +08:00
7e82e7651a
[Improve](txn) Add some fuzzy test stub in txn ( #26712 )
2023-11-16 11:50:06 +08:00
7fbc6d26a7
[debug](log) add some log to debug issue about insert ( #27045 )
2023-11-16 11:46:47 +08:00
10ee48bb6f
[fix](publish version) publish version task no need return VERSION_NOT_EXIST #27005
...
if BE's tablet not contains a txn, publish txn on them will no error, when check version exists it will indicate the tablet as error_tablet_id in task's response, so FE can know this tablet has fail.
Also for task, it's no need to set its status as "VERSION_NOT_EXIST". Because if set it as not ok, the BE will try this task two times. Since not contains this tablet's txn, the retry is in vain.
2023-11-15 21:09:54 +08:00
ebc15fc6cc
[fix](transaction) Fix concurrent schema change and txn cause dead lock ( #26428 )
...
Concurrent schema change and txn may cause dead lock. An example:
Txn T commit but not publish;
Run schema change or rollup on T's related partition, add alter replica R;
sc/rollup add a sched txn watermark M;
Restart fe;
After fe restart, T's loadedTblIndexes will clear because it's not save to disk;
T will publish version to all tablet, including sc/rollup's new alter replica R;
Since R not contains txn data, so the T will fail. It will then always waitting for R's data;
sc/rollup wait for txn before M to finish, only after that it will let R copy history data;
Since T's not finished, so sc/rollup will always wait, so R will nerver copy history data;
Txn T and sc/rollup will wait each other forever, cause dead lock;
Fix: because sc/rollup will ensure double write after the sched watermark M, so for finish transaction, when checking a alter replica:
if txn id is bigger than M, check it just like a normal replica;
otherwise skip check this replica, the BE will modify history data later.
2023-11-13 21:39:28 +08:00
c26f5a2bd2
[improvement](BE) Remove unnecessary error handling codes ( #26760 )
2023-11-12 00:02:51 +08:00
d767804815
[feature](merge-cloud) Decouple rowset id generator and local rowsets gc implementation ( #25921 )
2023-11-10 10:07:02 +08:00
bd2f007d52
[fix](load) Fix not check if the tablet is nullptr ( #26015 )
2023-10-29 10:01:14 +08:00
f31c1d858a
[fix](merge-on-write) fix duplicate key in schema change ( #25705 )
...
It should be ensured that the obtained versions are continuous when calculate delete bitmap calculations in publish.
The remaining NOTREADY tablet in the schema change failure should be dropped.
When a rowset was deleted, the delete bitmap cannot be deleted until there are no read requests to use the rowset.
2023-10-25 05:59:48 -05:00
9c9fc84f39
[feature](merge-cloud) Abstract BaseTablet for CloudTablet ( #24929 )
2023-10-18 20:29:04 +08:00
73c3e3ab55
[Feature](x-load) support config min replica num for loading data ( #21118 )
2023-10-11 21:07:35 +08:00
239df5860b
[enhancement](tablet_meta_lock) add more trace for write lock of tablet's _meta_lock ( #25095 )
2023-10-08 10:28:10 +08:00
642e5cdb69
[Fix](Status) Make Status [[nodiscard]] and handle returned Status correctly ( #23395 )
2023-09-29 22:38:52 +08:00
188d9ab94e
[enhancement](statistics) collect table level loaded rows on BE to make RPC light weight ( #24609 )
2023-09-28 10:51:50 +08:00
6502da8917
[bugfix](restore) add partition id into convert_rowset_ids() ( #24834 )
2023-09-25 20:07:24 +08:00
900b06d14e
[improvement](disk balancer) reduce disk balance sensitivity ( #24637 )
2023-09-24 22:02:53 +08:00
3bf95b2cfe
[fix](merge-on-write) fix duplicate key in schema change ( #24782 )
2023-09-22 19:50:31 +08:00
b092bdaabf
[feature](load) collect loaded rows on table level after txn published ( #24346 )
...
As title.
Stream load 20 lines
```
2023-09-14 11:40:04,186 DEBUG (PUBLISH_VERSION|23) [DatabaseTransactionMgr.updateCatalogAfterVisible():1769] table id to loaded rows:{51016=20}
```
```
mysql> select count(*) from dup_tbl_basic;
+----------+
| count(*) |
+----------+
| 20 |
+----------+
1 row in set (0.05 sec)
```
2023-09-19 12:00:08 +08:00
d20365cdcf
[fix](transaction) fix publish txn fake succ ( #24273 )
2023-09-14 21:04:59 +08:00
9c6734e68e
[bugfix](index) Fix build index limitations ( #24358 )
...
1. skip existed index on column with different id on build index
2. allow build index for CANCELED or FINISHED state
2023-09-14 17:53:22 +08:00
9d2fc78bd5
[fix](cooldown) Fix potential data loss when clone task's dst tablet is cooldown replica ( #17644 )
...
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com >
Co-authored-by: Kang <kxiao.tiger@gmail.com >
2023-09-01 15:27:52 +08:00
91c5640cae
[fix](tablet clone) fix clone backend chose wrong disk ( #23729 )
2023-09-01 15:12:35 +08:00
f7a3d2778a
[FIX](array)update array olapconvertor and support array nested other complex type ( #23489 )
...
* update array olapconvertor and support array nested other complex type
* update for inverted index
2023-08-29 16:18:11 +08:00
da9eb79ac4
[Enhancement](Schema hash) Remove schema hash in tablet info ( #23516 )
2023-08-29 10:05:12 +08:00
d4694167a8
[Enhancement](chore) Some Status relevant enhancement ( #23072 )
2023-08-21 14:14:38 +08:00
f0ad3ef244
[fix](merge-on-write) should use write lock of tablet's header lock in #23047 ( #23161 )
2023-08-18 17:50:44 +08:00
e6fe8c05d1
[fix](inverted index change) fix update delete bitmap incompletely when build inverted index on mow table ( #23047 )
2023-08-18 11:15:39 +08:00
de98324ea7
[fix](inverted index change) make mutex for ALTER_INVERTED_INDEX task and STORAGE_MEDIUM_MIGRATE task ( #22995 )
2023-08-18 08:35:30 +08:00
