218 Commits

Author SHA1 Message Date
c1abcf8e9c s3: split the common code of s3like store (#65602)
ref pingcap/tidb#65461
2026-01-16 08:45:44 +00:00
28cce6a472 objstore: move s3 like store into separate pkg (#65513)
ref pingcap/tidb#65461
2026-01-12 10:01:26 +00:00
e0df5fdab0 objstore: encapsulate i/o related logic into separate pkg and refactor (#65495)
ref pingcap/tidb#65461
2026-01-09 14:22:32 +00:00
607968304a lightning: add import into backend (#65321)
ref pingcap/tidb#65092
2026-01-09 01:05:35 +00:00
1e2cdaa16f objstore: move from 'br/pkg' to 'pkg/' directory (#65462)
ref pingcap/tidb#65461
2026-01-08 08:50:47 +00:00
1e1a19187a *: tidb txn supports to setting WithReturnCommitTS (#65306)
close pingcap/tidb#64537
2026-01-05 02:24:32 +00:00
c342953b2d dxf: rename the pkg from "disttask" to "dxf" (#65396)
ref pingcap/tidb#61702
2026-01-05 00:30:59 +00:00
514f19032d retry: make context.DeadlineExceeded retryable in local backend (#65345)
close pingcap/tidb#60637
2025-12-31 03:35:49 +00:00
8c3f0a502a backend: remove unexpected escape in external writer (#65316)
close pingcap/tidb#65319
2025-12-30 09:33:41 +00:00
b7bfd6c565 lightning: refactor and add import into checkpoint controller (#65241)
ref pingcap/tidb#65092
2025-12-29 17:57:02 +00:00
18eae64538 lightning: add import into checkpoint implementation (#65184)
ref pingcap/tidb#65092
2025-12-26 11:50:35 +00:00
5c06b05099 lightning: optimize forceTableSplitRange and skip for small tables (#65177)
close pingcap/tidb#65176
2025-12-26 10:42:25 +00:00
a5080263ff importinto: make ErrPDServerTimeout retryable which might meet during checksum (#65239)
close pingcap/tidb#65080
2025-12-24 12:50:53 +00:00
0eb881e406 importer: parallel split single large CSV files during generate import spec (#64875)
close pingcap/tidb#64810
2025-12-19 04:04:47 +00:00
23262123f0 br: upgrade AWS sdk from v1 to v2 (#64303)
close pingcap/tidb#62541
2025-12-17 08:43:20 +00:00
5e4e148f00 importinto: support conflict resolution in classic kernel (#65073)
ref pingcap/tidb#60621
2025-12-17 04:26:14 +00:00
5810fff4e5 *: ref all the jobs before sending to jobToWorkerCh (#64767)
close pingcap/tidb#64666
2025-12-14 06:47:47 +00:00
dc7e08f9c8 importinto: add helper methods and some type/enum definition for conflict resolve (#64956)
ref pingcap/tidb#60621
2025-12-11 06:35:02 +00:00
c49ac642c7 ddl/dxf: modify ingest param at runtime for add-index with global sort (#62120)
ref pingcap/tidb#57229, ref pingcap/tidb#57497
2025-12-08 10:47:14 +00:00
c5cd3e6a16 importinto: add bounded handle set for conflict resolution (#64844)
ref pingcap/tidb#60621
2025-12-04 06:59:41 +00:00
5e340159ca importinto/addindex: log object storage access and range/region job key count (#64650)
ref pingcap/tidb#61702
2025-11-24 15:03:32 +00:00
ee1732a381 retry: avoid log un-retryable error when it's nil or context cancelled (#64590)
ref pingcap/tidb#61702
2025-11-20 14:48:32 +00:00
d682970674 dupeDetector: enhance index conflict detection and logging for duplicate (#64513)
close pingcap/tidb#62285
2025-11-19 12:04:44 +00:00
4b2223cdd9 dxf: separate metered traffic into object storage and cluster and fix missing cluster traffic for import-into (#64523)
close pingcap/tidb#61702
2025-11-18 18:19:44 +00:00
c1247a0fc4 disttask: fix flaky summary collection for ingest step (#64471)
ref pingcap/tidb#61088
2025-11-17 09:31:05 +00:00
8e799f74e9 ingest: skip write when the region has no leader (#64442)
ref pingcap/tidb#61702, close pingcap/tidb#64422
2025-11-13 11:33:35 +00:00
9e38acfdbf dxf: make api request metering more accurate by intercepting request call (#64402)
ref pingcap/tidb#61702
2025-11-12 14:00:53 +00:00
82e2c6fd60 ddl/dxf: modify merge sort param at runtime for add-index with global sort (#62084)
ref pingcap/tidb#57229, ref pingcap/tidb#57497
2025-11-12 10:17:59 +00:00
779e298772 importinto/lightning: change library for parquet import (#63979)
ref pingcap/tidb#62906
2025-11-06 04:35:26 +00:00
f711b75a90 globalsort: export struct and fields to prepare for conflict resolution (#64283)
ref pingcap/tidb#60621
2025-11-05 15:03:14 +00:00
b2fc816077 backend: ensure engine cleanup to be executed (#64105)
close pingcap/tidb#64101
2025-10-29 11:14:16 +00:00
ee4eac2ccb global sort: update the part size calculate formula to avoid upload multipart failed (#63918)
close pingcap/tidb#63781
2025-10-21 03:19:49 +00:00
ad823cc1ff globalsort: add some test for one writer (#63972)
ref pingcap/tidb#60621
2025-10-15 11:07:31 +00:00
3dd656d7da importer: parallel init data source files (#63621)
close pingcap/tidb#62633
2025-10-15 10:18:47 +00:00
8ea80e6b2b lightning: increase source dir length (#63756)
close pingcap/tidb#63755
2025-09-25 12:33:46 +00:00
ef6f8e723e ingest: adapt AddPartitionRange api for ingest (#63467)
close pingcap/tidb#63466
2025-09-23 16:54:37 +00:00
a8ef81a436 lightning: return error if missing trailing semicolon for lightning schema file (#63415)
close pingcap/tidb#63414
2025-09-23 10:23:16 +00:00
7b166f6d1d localbackend: add some log to help diagnose (#63682)
ref pingcap/tidb#63681
2025-09-23 05:13:31 +00:00
eb25197d68 disttask: collect metering data to subtask's real time summary (#63431)
ref pingcap/tidb#61702
2025-09-15 12:47:09 +00:00
1f967030a0 fix: drop some dead code phase 1 (#63079)
ref pingcap/tidb#50727
2025-09-05 09:54:33 +00:00
bd5c194581 ImportInto: add import sdk for cloud (#61545)
ref pingcap/tidb#61264
2025-09-05 03:48:12 +00:00
cb95b10975 lightning, importer: determine whether to skip reading parquet based on the table structure (#62994)
close pingcap/tidb#63254
2025-09-04 07:58:21 +00:00
ba8710bea9 backend/local: add missing collector for next-gen ingest (#63178)
ref pingcap/tidb#61088
2025-08-27 04:44:28 +00:00
4f18d8d4b1 importinto: set request resource during checksum for nextgen (#63155)
ref pingcap/tidb#61702
2025-08-25 15:29:34 +00:00
fdca8155d1 global sort: reduce the memory usage of merge sort concurrent reader (#62921)
close pingcap/tidb#62853
2025-08-21 05:19:20 +00:00
6361402965 server/handler: register missing http handlers related to ingest param (#63099)
ref pingcap/tidb#61553
2025-08-21 03:41:29 +00:00
8c88b61875 tests: fix unit tests for next-gen (#62895)
ref pingcap/tidb#61702
2025-08-20 19:51:02 +00:00
3b96fe2123 perf: using generics instead of reflect for better performance (#62973)
close pingcap/tidb#62972
2025-08-20 18:04:01 +00:00
92e64b2699 global_sort: fix incorrect write speed metrics in merge-sort step (#62847)
close pingcap/tidb#62870
2025-08-14 15:39:51 +00:00
b529ae5267 globalsort: use partitioned prefix to store intermediate files (#62905)
close pingcap/tidb#62904
2025-08-12 07:08:42 +00:00