155 Commits

Author SHA1 Message Date
c1abcf8e9c s3: split the common code of s3like store (#65602)
ref pingcap/tidb#65461
2026-01-16 08:45:44 +00:00
28cce6a472 objstore: move s3 like store into separate pkg (#65513)
ref pingcap/tidb#65461
2026-01-12 10:01:26 +00:00
e0df5fdab0 objstore: encapsulate i/o related logic into separate pkg and refactor (#65495)
ref pingcap/tidb#65461
2026-01-09 14:22:32 +00:00
1e2cdaa16f objstore: move from 'br/pkg' to 'pkg/' directory (#65462)
ref pingcap/tidb#65461
2026-01-08 08:50:47 +00:00
1e1a19187a *: tidb txn supports to setting WithReturnCommitTS (#65306)
close pingcap/tidb#64537
2026-01-05 02:24:32 +00:00
c342953b2d dxf: rename the pkg from "disttask" to "dxf" (#65396)
ref pingcap/tidb#61702
2026-01-05 00:30:59 +00:00
8c3f0a502a backend: remove unexpected escape in external writer (#65316)
close pingcap/tidb#65319
2025-12-30 09:33:41 +00:00
5c06b05099 lightning: optimize forceTableSplitRange and skip for small tables (#65177)
close pingcap/tidb#65176
2025-12-26 10:42:25 +00:00
23262123f0 br: upgrade AWS sdk from v1 to v2 (#64303)
close pingcap/tidb#62541
2025-12-17 08:43:20 +00:00
5e4e148f00 importinto: support conflict resolution in classic kernel (#65073)
ref pingcap/tidb#60621
2025-12-17 04:26:14 +00:00
5810fff4e5 *: ref all the jobs before sending to jobToWorkerCh (#64767)
close pingcap/tidb#64666
2025-12-14 06:47:47 +00:00
dc7e08f9c8 importinto: add helper methods and some type/enum definition for conflict resolve (#64956)
ref pingcap/tidb#60621
2025-12-11 06:35:02 +00:00
c49ac642c7 ddl/dxf: modify ingest param at runtime for add-index with global sort (#62120)
ref pingcap/tidb#57229, ref pingcap/tidb#57497
2025-12-08 10:47:14 +00:00
c5cd3e6a16 importinto: add bounded handle set for conflict resolution (#64844)
ref pingcap/tidb#60621
2025-12-04 06:59:41 +00:00
5e340159ca importinto/addindex: log object storage access and range/region job key count (#64650)
ref pingcap/tidb#61702
2025-11-24 15:03:32 +00:00
d682970674 dupeDetector: enhance index conflict detection and logging for duplicate (#64513)
close pingcap/tidb#62285
2025-11-19 12:04:44 +00:00
4b2223cdd9 dxf: separate metered traffic into object storage and cluster and fix missing cluster traffic for import-into (#64523)
close pingcap/tidb#61702
2025-11-18 18:19:44 +00:00
c1247a0fc4 disttask: fix flaky summary collection for ingest step (#64471)
ref pingcap/tidb#61088
2025-11-17 09:31:05 +00:00
8e799f74e9 ingest: skip write when the region has no leader (#64442)
ref pingcap/tidb#61702, close pingcap/tidb#64422
2025-11-13 11:33:35 +00:00
9e38acfdbf dxf: make api request metering more accurate by intercepting request call (#64402)
ref pingcap/tidb#61702
2025-11-12 14:00:53 +00:00
82e2c6fd60 ddl/dxf: modify merge sort param at runtime for add-index with global sort (#62084)
ref pingcap/tidb#57229, ref pingcap/tidb#57497
2025-11-12 10:17:59 +00:00
f711b75a90 globalsort: export struct and fields to prepare for conflict resolution (#64283)
ref pingcap/tidb#60621
2025-11-05 15:03:14 +00:00
b2fc816077 backend: ensure engine cleanup to be executed (#64105)
close pingcap/tidb#64101
2025-10-29 11:14:16 +00:00
ee4eac2ccb global sort: update the part size calculate formula to avoid upload multipart failed (#63918)
close pingcap/tidb#63781
2025-10-21 03:19:49 +00:00
ad823cc1ff globalsort: add some test for one writer (#63972)
ref pingcap/tidb#60621
2025-10-15 11:07:31 +00:00
ef6f8e723e ingest: adapt AddPartitionRange api for ingest (#63467)
close pingcap/tidb#63466
2025-09-23 16:54:37 +00:00
7b166f6d1d localbackend: add some log to help diagnose (#63682)
ref pingcap/tidb#63681
2025-09-23 05:13:31 +00:00
eb25197d68 disttask: collect metering data to subtask's real time summary (#63431)
ref pingcap/tidb#61702
2025-09-15 12:47:09 +00:00
ba8710bea9 backend/local: add missing collector for next-gen ingest (#63178)
ref pingcap/tidb#61088
2025-08-27 04:44:28 +00:00
4f18d8d4b1 importinto: set request resource during checksum for nextgen (#63155)
ref pingcap/tidb#61702
2025-08-25 15:29:34 +00:00
fdca8155d1 global sort: reduce the memory usage of merge sort concurrent reader (#62921)
close pingcap/tidb#62853
2025-08-21 05:19:20 +00:00
6361402965 server/handler: register missing http handlers related to ingest param (#63099)
ref pingcap/tidb#61553
2025-08-21 03:41:29 +00:00
92e64b2699 global_sort: fix incorrect write speed metrics in merge-sort step (#62847)
close pingcap/tidb#62870
2025-08-14 15:39:51 +00:00
b529ae5267 globalsort: use partitioned prefix to store intermediate files (#62905)
close pingcap/tidb#62904
2025-08-12 07:08:42 +00:00
6f38f974a3 tests: fix realtikv test for next-gen (#62701)
close pingcap/tidb#60688, ref pingcap/tidb#61702
2025-08-06 08:14:15 +00:00
a9c4d868f5 *: add merge speed for global sort (#62550)
ref pingcap/tidb#61088
2025-07-25 05:57:13 +00:00
06674e2309 *: adjusts the merge sort overlap threshold/merge sort file count step based on concurrency (#62483)
close pingcap/tidb#60951
2025-07-24 17:01:07 +00:00
f2534055f2 ddl: send result chunks to writers after scan task complete (#62014)
close pingcap/tidb#62008
2025-07-18 12:01:49 +00:00
3606de5c43 backend: add log fields for local and external pkg (#62476)
ref pingcap/tidb#61702
2025-07-18 05:49:42 +00:00
65c3cf222d globalsort: fix stuck when ingesting small kv (#62407)
close pingcap/tidb#62388
2025-07-15 04:37:06 +00:00
5ec4f36c85 dxf: use correct store in add-index, checksum by copr in import-into, and enable DXF service (#62224)
ref pingcap/tidb#61702
2025-07-09 08:07:27 +00:00
86ed1cab62 backend/local: add rate limiter for split region and ingest data (#61555)
close pingcap/tidb#61553
2025-07-05 09:30:09 +00:00
0a704a4bb7 *: collect subtask summary for IMPORT INTO tasks (#61600)
ref pingcap/tidb#61088
2025-07-04 15:12:30 +00:00
6106300325 localbackend: release the service safe point on finish checksum (#62143)
ref pingcap/tidb#61702, close pingcap/tidb#62142
2025-07-02 13:45:42 +00:00
02757471f1 statistics, lightning: remove useless code (#62004) 2025-06-26 03:07:36 +00:00
8123a22d38 importinto: some prepare work for conflict resolve (#61426)
ref pingcap/tidb#60621
2025-06-10 12:21:59 +00:00
a165d9fd7c import: adjust data mem buf block size based on max_entry_size_limit(#60139) (#61399)
close pingcap/tidb#61398
2025-06-09 03:38:43 +00:00
cc0925eeaf Lightning: Attempt to return writeTooSlow when we experience write timeout. (#61346)
ref pingcap/tidb#61326
2025-06-06 07:09:45 +00:00
d216b7be1a ingestor: fix reused httpclient schema and move ingest api err to ingestcli pkg (#61492)
ref pingcap/tidb#60418
2025-06-05 13:10:44 +00:00
029c0a90d0 *: inject random error for global sort & injest (#60582)
close pingcap/tidb#60277
2025-06-04 11:29:23 +00:00