Commit Graph

228 Commits

Author SHA1 Message Date
78320d3169 dxf: separate runtime slots(concurrency) out of required slot to workaround OOM (#65265)
ref pingcap/tidb#65192
2025-12-26 08:26:07 +00:00
62ee49c7b1 dxf: rename task concurrency to required slots and other refactor (#65195)
ref pingcap/tidb#65192
2025-12-25 05:07:47 +00:00
9ee409cc4a dxf: add event metric for cleanup or meter failure so we can add alert (#65124)
ref pingcap/tidb#61702
2025-12-19 09:06:41 +00:00
865730dd6e dxf: retry failed write with the same TS instead of accumulating meter data to avoid duplicate data (#65115)
ref pingcap/tidb#61702
2025-12-18 18:18:17 +00:00
6a1556e06e dxf: add concurrency/max-node/duration to the final metering data to help estimate cost (#65039)
ref pingcap/tidb#61702
2025-12-16 18:41:47 +00:00
dc7e08f9c8 importinto: add helper methods and some type/enum definition for conflict resolve (#64956)
ref pingcap/tidb#60621
2025-12-11 06:35:02 +00:00
c49ac642c7 ddl/dxf: modify ingest param at runtime for add-index with global sort (#62120)
ref pingcap/tidb#57229, ref pingcap/tidb#57497
2025-12-08 10:47:14 +00:00
45b1597860 dxf: fix meter data overflow if we register recorder of the same task again after unregister (#64861)
ref pingcap/tidb#61702
2025-12-05 07:29:38 +00:00
be3d5b4b0f *: flight recorder tracing for internal sessions (#64341)
ref pingcap/tidb#64008
2025-11-21 06:03:19 +00:00
6eeb23e317 add-index/import-into: meter row and kv size data and disable some options for import-into in nextgen (#64584)
ref pingcap/tidb#61702
2025-11-20 16:29:44 +00:00
17945b9a23 ingest: fix default region split size and keys in nextgen (#64540)
ref pingcap/tidb#61702, close pingcap/tidb#64526
2025-11-19 06:16:34 +00:00
4b2223cdd9 dxf: separate metered traffic into object storage and cluster and fix missing cluster traffic for import-into (#64523)
close pingcap/tidb#61702
2025-11-18 18:19:44 +00:00
c1247a0fc4 disttask: fix flaky summary collection for ingest step (#64471)
ref pingcap/tidb#61088
2025-11-17 09:31:05 +00:00
1a04b35a60 importinto: fix flaky test TestAutoDetectFileType (#64478)
close pingcap/tidb#64106
2025-11-14 06:53:38 +00:00
9e38acfdbf dxf: make api request metering more accurate by intercepting request call (#64402)
ref pingcap/tidb#61702
2025-11-12 14:00:53 +00:00
82e2c6fd60 ddl/dxf: modify merge sort param at runtime for add-index with global sort (#62084)
ref pingcap/tidb#57229, ref pingcap/tidb#57497
2025-11-12 10:17:59 +00:00
f2abf53fab ddl: add checkpoint operator for DXF local add index (#64171)
ref pingcap/tidb#62285
2025-11-11 10:58:02 +00:00
63873b4c50 dxf: fix metering might be lost when failed to write, and refactor (#64358)
ref pingcap/tidb#61702
2025-11-10 13:48:17 +00:00
5541db5f4b dxf: fix incorrect worker count metric when total task count exceed schedule limit (#64321)
ref pingcap/tidb#61702
2025-11-06 13:31:29 +00:00
968e31fc3f ddl: cancel the job context before rolling back (#64130)
close pingcap/tidb#64129
2025-10-31 01:47:37 +00:00
bdd2b6f284 dxf/ddl: always enable DXF and fast reorg in nextgen (#64179)
ref pingcap/tidb#61702
2025-10-30 17:24:13 +00:00
9dc8189afa dxf: add metrics for alert purpose in nextgen (#64100)
ref pingcap/tidb#61702
2025-10-27 07:27:31 +00:00
50a1e5a631 dxf: fix task cannot be cancelled when the number of active scheduler reached the limit (#63897)
ref pingcap/tidb#61702, close pingcap/tidb#63896
2025-10-11 03:38:21 +00:00
b6e32dd757 ddl: run merge-temp-index step on DXF (#62971)
close pingcap/tidb#63296
2025-10-03 10:15:04 +00:00
b3011b689d disttask: support metering on next-gen (#63397)
ref pingcap/tidb#61702
2025-09-19 02:27:35 +00:00
276ed0cf95 dxf: add api to manually tune the calculation of resource related params (#63517)
ref pingcap/tidb#61702
2025-09-18 15:09:53 +00:00
eb25197d68 disttask: collect metering data to subtask's real time summary (#63431)
ref pingcap/tidb#61702
2025-09-15 12:47:09 +00:00
941ba546bd importinto: avoid cal resource params twice and round those params (#63452)
ref pingcap/tidb#61702
2025-09-11 08:19:15 +00:00
37f8a14657 importinto: fix zero update time during validation phase and refine logs (#63425)
ref pingcap/tidb#61702
2025-09-09 12:07:54 +00:00
9825dfebc9 importinto: calculate distsql scan concurency in nextgen (#63341)
ref pingcap/tidb#61702
2025-09-03 11:52:06 +00:00
36a2af4b10 dxf: start scheduler without reserve resource in nextgen (#63205)
ref pingcap/tidb#61702
2025-08-26 12:22:10 +00:00
97c5139bf9 dxf: add schedule status api for nextgen (#63159)
ref pingcap/tidb#61702
2025-08-25 10:45:07 +00:00
1fa4788853 dxf: prepare work for schedule status API in nextgen (#63140)
ref pingcap/tidb#61702
2025-08-22 17:21:38 +00:00
aa875ea472 dxf: calculate max_node_count and concurrency for next-gen automatically (#62995)
ref pingcap/tidb#61702
2025-08-22 14:53:38 +00:00
658fa1a43e crossks: add real-tikv tests for info schema sync (#63008)
ref pingcap/tidb#61702
2025-08-19 17:29:50 +00:00
36dc5f0871 dxf: support modify max_node_count param (#62999)
ref pingcap/tidb#61702
2025-08-19 04:21:02 +00:00
d82d2baa0f bazel: update settings and close some race on tests (#63009) 2025-08-15 09:07:31 +00:00
9cd2b03833 dxf/crossks: check by inner fields not global var and make crossks real tikvtest work (#62918)
ref pingcap/tidb#61702
2025-08-14 15:00:03 +00:00
9b12530970 crossks: join online schema change server group and sync info schema (#62860)
ref pingcap/tidb#61702
2025-08-13 04:51:06 +00:00
2a0741939c disttask: refine the speed calculation of import jobs (#62638)
ref pingcap/tidb#61088
2025-08-09 01:15:27 +00:00
eb12403896 *: replace make then copy to slices.Clone (#62544)
close pingcap/tidb#62543
2025-07-31 12:27:55 +00:00
1ea1db40cd infosync: move server info sync part of structs out to a separate pkg (#62609)
ref pingcap/tidb#61702
2025-07-25 07:46:59 +00:00
5ae6ad63c2 session: rename and unify location of interfaces, and some refactor (#62498)
ref pingcap/tidb#61702
2025-07-22 07:43:38 +00:00
8060477c03 importinto: reduce def engine size to 100G and always use max node count to adjust subtask count (#62527)
ref pingcap/tidb#61702
2025-07-21 07:41:31 +00:00
0fc2d8bfd8 tests: add create index realtikv test for next_gen (#62326)
ref pingcap/tidb#61702
2025-07-18 07:16:24 +00:00
4048d4098b dxf: fix flaky test TestResourceControl/scale-in (#62489)
close pingcap/tidb#61354
2025-07-18 05:49:48 +00:00
3606de5c43 backend: add log fields for local and external pkg (#62476)
ref pingcap/tidb#61702
2025-07-18 05:49:42 +00:00
5ec4f36c85 dxf: use correct store in add-index, checksum by copr in import-into, and enable DXF service (#62224)
ref pingcap/tidb#61702
2025-07-09 08:07:27 +00:00
0a704a4bb7 *: collect subtask summary for IMPORT INTO tasks (#61600)
ref pingcap/tidb#61088
2025-07-04 15:12:30 +00:00
c56481e1d1 ddl: run add-index subtasks at system keyspace tidb (#62045)
ref pingcap/tidb#61702
2025-07-02 10:38:25 +00:00