Commit Graph

236 Commits

Author SHA1 Message Date
5ec4f36c85 dxf: use correct store in add-index, checksum by copr in import-into, and enable DXF service (#62224)
ref pingcap/tidb#61702
2025-07-09 08:07:27 +00:00
0a704a4bb7 *: collect subtask summary for IMPORT INTO tasks (#61600)
ref pingcap/tidb#61088
2025-07-04 15:12:30 +00:00
b8a0d7f075 importinto: encode and ingest step use task keyspace in nextgen (#62138)
ref pingcap/tidb#61702
2025-07-03 17:46:45 +00:00
c56481e1d1 ddl: run add-index subtasks at system keyspace tidb (#62045)
ref pingcap/tidb#61702
2025-07-02 10:38:25 +00:00
6e13f827fc realtikvtest: make import-into/add-index tests works on nextgen (#62112)
ref pingcap/tidb#61702
2025-07-01 15:58:33 +00:00
a90774c2e5 testkit: make ut works in nextgen after introduce SYSTEM ks (#62113)
ref pingcap/tidb#61702
2025-07-01 10:20:55 +00:00
ae70341a13 disttask, util: remove useless code (#62056) 2025-06-30 07:05:18 +00:00
704373b1e4 importinto: use correct session to submit/update/query task in DXF service (#62023)
ref pingcap/tidb#61702
2025-06-28 06:07:21 +00:00
ffaec74aed disttask: add keyspace name to task_key for next-gen (#61860)
ref pingcap/tidb#61702
2025-06-22 13:12:39 +00:00
8e6d77ae27 dxf: add keyspace name to global task table (#61755)
ref pingcap/tidb#61702
2025-06-18 16:41:39 +00:00
8123a22d38 importinto: some prepare work for conflict resolve (#61426)
ref pingcap/tidb#60621
2025-06-10 12:21:59 +00:00
292c2c5caf global sort: use cluster id in file path (#61502)
close pingcap/tidb#61537
2025-06-10 06:34:40 +00:00
254849d23a ddl, disttask: extend SubtaskSummary and add Collector interface (#61425)
ref pingcap/tidb#61088
2025-06-09 08:12:37 +00:00
a165d9fd7c import: adjust data mem buf block size based on max_entry_size_limit(#60139) (#61399)
close pingcap/tidb#61398
2025-06-09 03:38:43 +00:00
0f5aa223ec *: rewrite InTest to EnableInternalCheck (#60167)
ref pingcap/tidb#60176
2025-06-05 07:56:08 +00:00
c1a4550fd4 dxf: make target scope fixed to dxf_service on nextgen (#61427)
ref pingcap/tidb#60418
2025-06-04 06:53:02 +00:00
2036543b29 disttask: set moderate timeout for pkg/disttask/framework/storage (#61463)
close pingcap/tidb#61461
2025-06-03 09:26:38 +00:00
5a00a10d38 disttask: distribute merge sort subtasks to all available nodes (#60395)
close pingcap/tidb#60375
2025-05-24 04:21:07 +00:00
fe769f5154 dxf: include context info in the stored error (#61289)
close pingcap/tidb#60433
2025-05-23 13:28:13 +00:00
c71ebaf73e global-sort: accelerate adding unique index using global sort (#60848)
ref pingcap/tidb#60621
2025-05-12 09:25:58 +00:00
4ad4493246 ddl/dxf: modify param at runtime for add-index with global sort (part 1) (#60677)
ref pingcap/tidb#57229, ref pingcap/tidb#57497
2025-04-29 05:22:19 +00:00
dc876e5c51 disttask: gopls modernize (#60795)
close pingcap/tidb#60794
2025-04-28 10:06:26 +00:00
0200e5d634 localbackend: merge feature branch next-gen (#60879)
ref pingcap/tidb#60418
2025-04-28 05:18:58 +00:00
5faa7496ef ddl: fix OOM issue when adding many indexes (#60676)
ref pingcap/tidb#59508
2025-04-25 08:25:35 +00:00
531eb80580 *: add a RetryableErrorCount metric to trace error during DDL (#60485)
ref pingcap/tidb#60484
2025-04-22 14:23:02 +00:00
08d20b8ee2 IMPORTINTO: close store inside LoadDataController (#60633)
close pingcap/tidb#60352
2025-04-18 11:18:52 +00:00
813b32f4df import: add a hidden option to enable manual recover for debug purpose (#59674)
ref pingcap/tidb#59640
2025-04-08 08:46:48 +00:00
1cd39430e1 *: inject random error for DXF (part2) (#60354)
ref pingcap/tidb#60277
2025-04-08 06:18:54 +00:00
f6f5703e43 importinto: fix error not returned when writing external meta (#60350)
close pingcap/tidb#60083
2025-04-01 09:04:15 +00:00
6e5a6d2a93 importinto: check retryable schedule error (#60320)
ref pingcap/tidb#59754
2025-03-28 11:50:24 +00:00
e2d1d89c9e *: inject random error for DXF (part1) (#60278)
ref pingcap/tidb#60277
2025-03-27 12:38:13 +00:00
ed8b869a60 globalsort: write sorted kv meta to external storage (#59966)
close pingcap/tidb#60083
2025-03-26 09:39:54 +00:00
b08a92082f disttask: fix task metrics for distributed execute framework panel (#59940)
ref pingcap/tidb#47017
2025-03-25 17:31:28 +00:00
d51e00e5bb globalsort: reduce number of SST ingested into TiKV (#59870) (#60045)
ref pingcap/tidb#51734, close pingcap/tidb#59962
2025-03-24 12:22:43 +00:00
80d6b5683c infosync: refactor server config into dynamic and static sections (#58473)
close pingcap/tidb#58405
2025-03-18 02:36:52 +00:00
6953d3656c disttask: fix potential block on DXF scheduler (#59945)
close pingcap/tidb#59944
2025-03-14 05:41:34 +00:00
246a241621 dxf: add manual recovery mode (#59641)
ref pingcap/tidb#59640
2025-02-24 09:42:51 +00:00
e2d5d1c017 import-into: support running with 1 thread when using global sort (#58572)
close pingcap/tidb#58680
2025-02-24 06:30:23 +00:00
811be5aaa5 disttask: add local disk space to node resource managed by DXF (#59607)
ref pingcap/tidb#49008
2025-02-20 08:42:09 +00:00
b45a9e840c sysvar: add validation check for tidb_max_dist_task_nodes (#59645)
ref pingcap/tidb#58944
2025-02-20 07:05:27 +00:00
54384b7cbb ddl: modify param at runtime for add-index on DXF local sort (#59343)
ref pingcap/tidb#57229, ref pingcap/tidb#57497
2025-02-19 13:43:09 +00:00
9099090462 dxf: add tidb_max_dist_task_nodes to specify max node count (#58937)
close pingcap/tidb#58944
2025-02-18 05:12:25 +00:00
baf16005e0 ddl: part 1 of add-index integrating dxf dynamic modify param (#59046)
ref pingcap/tidb#57229, ref pingcap/tidb#57497
2025-02-17 05:19:39 +00:00
0c61a62c3d dxf: add integration test for modify param during subtask running (#59472)
ref pingcap/tidb#57497
2025-02-14 10:06:40 +00:00
fc0adfcf28 lightning/global-sort: move membuf pkg from br to lightning (#59550)
ref pingcap/tidb#57275
2025-02-14 09:19:19 +00:00
435a716670 import: fix oom on global sort when the target table has large number of indices (#59518)
close pingcap/tidb#59508
2025-02-14 06:13:19 +00:00
27f8ff3840 dxf: use global entry_size_limit when executing SQLs (#59516)
close pingcap/tidb#59506
2025-02-13 11:22:23 +00:00
fbdaef5c0b dxf: optimize querying eligible tasks in task executor manager (#59345)
close pingcap/tidb#59344
2025-02-10 07:33:52 +00:00
cf4d252af8 *: split definition out of variable pkg and refactor some reorg methods (#59146)
ref pingcap/tidb#57229, ref pingcap/tidb#57497
2025-01-23 09:24:44 +00:00
fcdbbd49bb DXF: check the cause of subtaskCtx to decide the subtask state (#58893)
close pingcap/tidb#58827
2025-01-22 16:20:26 +00:00