Commit Graph

29 Commits

Author SHA1 Message Date
f46db203f9 *: caller can use a fixed TS for fast reorg (#52993)
close pingcap/tidb#46986, ref pingcap/tidb#49233
2024-05-10 13:31:09 +00:00
72e5460ee8 lightning/importinto: fix insert err after import for AUTO_ID_CACHE=1 and SHARD_ROW_ID_BITS (#52712)
close pingcap/tidb#52654
2024-04-18 08:10:38 +00:00
76abd8b836 disttask,ddl: use subtask concurrency (#52286)
ref pingcap/tidb#49008
2024-04-02 04:24:16 +00:00
f2ea18c07f lightning: move lightning directory out of br (#52247)
ref pingcap/tidb#52182
2024-04-01 07:13:46 +00:00
ad63eb8d1e disttask: add StepExecFrameworkInfo as an example of Impl access Base (#52192)
ref pingcap/tidb#47006
2024-03-29 04:20:19 +00:00
490e17c9ab importinto: add log for chunk checksum & replace codec with keyspace (#52054)
ref pingcap/tidb#49008
2024-03-25 05:45:46 +00:00
df79e5ca11 globalsort: opt merge step mem to run on 8c (#51924)
ref pingcap/tidb#50752
2024-03-22 10:36:50 +00:00
61c471487e globalsort: split files evenly in MergeOverlappingFiles (#51878)
ref pingcap/tidb#50752
2024-03-19 08:41:13 +00:00
5a2b8e8e1a importinto: pass kv store directly when rebase ids (#51850)
close pingcap/tidb#51848
2024-03-18 07:54:42 +00:00
911bcf16a8 importinto: calculate writer mem size using step resource & add a option for test (#51486)
ref pingcap/tidb#49008
2024-03-07 10:59:06 +00:00
4182a418c8 *: save kv.Storage in import disttask components and use it (#51336)
ref pingcap/tidb#51337
2024-02-29 07:42:31 +00:00
332086ebc8 importinto: remove separate wait routine in detached mode and refactor (#51335)
ref pingcap/tidb#49008
2024-02-27 09:05:09 +00:00
23190ff540 disttask: enlarge max-concurrent-task and optimize scheduler/executor loop (#51241)
ref pingcap/tidb#49008
2024-02-27 03:32:01 +00:00
65d57c0a43 import into: log checksum for each KV group (#50871)
ref pingcap/tidb#50832
2024-02-18 08:55:55 +00:00
ba778070a2 external: onefile writer should have onClose, and add KV count to WriteSummary (#50955)
ref pingcap/tidb#50832
2024-02-04 09:32:43 +00:00
cac449b337 ddl, disttask: collect summary for each subtask (#50838)
ref pingcap/tidb#48795
2024-02-01 07:58:54 +00:00
b5542c5ed2 disttask: decouple handleExecutableTask from manager & separate executor context (#50724)
ref pingcap/tidb#49008
2024-01-30 05:22:52 +00:00
7ce7fb2a85 external: round robin to use multiple GCS clients (#50408)
close pingcap/tidb#50413
2024-01-23 09:43:49 +00:00
d1e6f1c291 disttask: remove revert subtask (#50550)
close pingcap/tidb#50549
2024-01-22 10:25:49 +00:00
8bb4d2f412 disttask: move step2Str to framework (#50565)
ref pingcap/tidb#49008
2024-01-22 03:12:19 +00:00
890230e17e disttask: add available resource param to step executor (#50475)
ref pingcap/tidb#49008
2024-01-17 11:51:16 +00:00
17d65b9317 disttask: change extra running subtasks to pending (#50266)
ref pingcap/tidb#49008
2024-01-12 03:48:25 +00:00
e15c517afa import/index: adjust concurrency using cpu-count of managed node depends on use case (#50091)
ref pingcap/tidb#49008
2024-01-10 08:31:54 +00:00
84d9bc3b6a importinto: move post-process code from disttask pkg to importer pkg (#49884)
ref pingcap/tidb#49883
2023-12-29 07:27:35 +00:00
da460f11b0 global sort: fix broken prop with onefile writer (#49589)
close pingcap/tidb#49590
2023-12-20 08:54:23 +00:00
323b0f772b *: remove deprecated LOAD DATA actions (#49470)
ref pingcap/tidb#44078
2023-12-15 14:00:21 +00:00
64430c359e disttask: rename dispatcher to scheduler (#49182)
close pingcap/tidb#49125
2023-12-13 03:33:22 +00:00
28cb579488 disttask: add IsRetryableError interface for taskExecutor (#49097)
ref pingcap/tidb#48795
2023-12-04 08:22:22 +00:00
b21bb3e7a7 disttask: rename scheduler to TaskExecutor (#49025)
ref pingcap/tidb#48795
2023-12-01 11:58:19 +00:00