64 Commits

Author SHA1 Message Date
28cce6a472 objstore: move s3 like store into separate pkg (#65513)
ref pingcap/tidb#65461
2026-01-12 10:01:26 +00:00
e0df5fdab0 objstore: encapsulate i/o related logic into separate pkg and refactor (#65495)
ref pingcap/tidb#65461
2026-01-09 14:22:32 +00:00
607968304a lightning: add import into backend (#65321)
ref pingcap/tidb#65092
2026-01-09 01:05:35 +00:00
1e2cdaa16f objstore: move from 'br/pkg' to 'pkg/' directory (#65462)
ref pingcap/tidb#65461
2026-01-08 08:50:47 +00:00
5269783b55 importer: fix flaky test (#65373)
close pingcap/tidb#53820
2026-01-07 13:41:50 +00:00
09ef45f7f8 lightning: fix logged content of "Sample source data" (#65340)
close pingcap/tidb#65337
2026-01-07 12:52:47 +00:00
b7bfd6c565 lightning: refactor and add import into checkpoint controller (#65241)
ref pingcap/tidb#65092
2025-12-29 17:57:02 +00:00
18eae64538 lightning: add import into checkpoint implementation (#65184)
ref pingcap/tidb#65092
2025-12-26 11:50:35 +00:00
6b3b545c6d lightning: skip TestLocalTempKVDirCheckBasic in nextgen (#65287)
close pingcap/tidb#65224
2025-12-26 10:43:06 +00:00
0eb881e406 importer: parallel split single large CSV files during generate import spec (#64875)
close pingcap/tidb#64810
2025-12-19 04:04:47 +00:00
779e298772 importinto/lightning: change library for parquet import (#63979)
ref pingcap/tidb#62906
2025-11-06 04:35:26 +00:00
ef6f8e723e ingest: adapt AddPartitionRange api for ingest (#63467)
close pingcap/tidb#63466
2025-09-23 16:54:37 +00:00
eb25197d68 disttask: collect metering data to subtask's real time summary (#63431)
ref pingcap/tidb#61702
2025-09-15 12:47:09 +00:00
cb95b10975 lightning, importer: determine whether to skip reading parquet based on the table structure (#62994)
close pingcap/tidb#63254
2025-09-04 07:58:21 +00:00
f76844ee28 test: fix some failed tests in nextgen (#63357)
ref pingcap/tidb#61702
2025-09-03 10:03:42 +00:00
d82d2baa0f bazel: update settings and close some race on tests (#63009) 2025-08-15 09:07:31 +00:00
3606de5c43 backend: add log fields for local and external pkg (#62476)
ref pingcap/tidb#61702
2025-07-18 05:49:42 +00:00
7b104b729a lightning: fix panic on checksum manager close when checksum is off (#62300)
close pingcap/tidb#62142
2025-07-08 11:53:24 +00:00
6106300325 localbackend: release the service safe point on finish checksum (#62143)
ref pingcap/tidb#61702, close pingcap/tidb#62142
2025-07-02 13:45:42 +00:00
480e7b454b Lightning: clarify disk-quota precheck message and add warning for invalid quota (#61818)
close pingcap/tidb#61542
2025-06-24 16:44:42 +00:00
f701e5ae3d dxfservice: create store for SYSTEM keyspace (#61752)
ref pingcap/tidb#61702
2025-06-24 07:10:15 +00:00
1a1897ec9c lightning: gopls modernize (#60768)
close pingcap/tidb#60767
2025-05-08 03:57:43 +00:00
593f552701 Lightning: Add RealOffset to checkpoint (#60723)
close pingcap/tidb#60734
2025-04-25 09:57:58 +00:00
48e5bdd13f Lightning: Fix grammar and spacing in Lightning error message (#60703)
close pingcap/tidb#60704
2025-04-22 13:11:32 +00:00
cc8d9cbbd4 lignthing/importinto: parallel reading files infos from data store (#59382)
close pingcap/tidb#56104, close pingcap/tidb#60224
2025-04-11 07:56:42 +00:00
f3d0e43040 *: upgrade to the latest client-go (#59757)
ref pingcap/tidb#58780
2025-03-27 07:32:36 +00:00
514204ea91 Lightning: Make sure we are using default block size of 16KB if user does not specify one. (#60097)
close pingcap/tidb#59947
2025-03-20 05:57:42 +00:00
6f49d25c70 br/lightning: Apply pd_enable_follower_handle_region to Lightning/BR (#59783)
close pingcap/tidb#58651
2025-03-03 11:12:31 +00:00
81995d6892 csv-parser: rename fields to better reflect its meaning (#59360)
ref pingcap/tidb#44893, ref pingcap/tidb#57275
2025-02-11 05:25:19 +00:00
a83ed8f7dd lightning: redact external storage url (#59256)
close pingcap/tidb#59086
2025-02-10 11:20:53 +00:00
cf4d252af8 *: split definition out of variable pkg and refactor some reorg methods (#59146)
ref pingcap/tidb#57229, ref pingcap/tidb#57497
2025-01-23 09:24:44 +00:00
b2f2faaa95 parser: move 'model' to 'ast' pkg (#58704)
ref pingcap/tidb#54436
2025-01-07 10:20:04 +00:00
14a469ab9d *: upgrade the PD client to adopt the latest refactor (#58440)
ref pingcap/tidb#58439
2024-12-23 10:36:12 +00:00
aec0fc58bd lightning: calculate concurrency of schema import from region-concurrency (#58202)
close pingcap/tidb#58141
2024-12-17 04:50:15 +00:00
ca395fa4be lightning: add precheck that PD and TiDB must belongs to same cluster (#57709)
close pingcap/tidb#57704
2024-11-27 13:18:17 +00:00
ecca340037 lightning: fix id too large after parallel import (#57398)
close pingcap/tidb#56814
2024-11-19 11:03:32 +00:00
zzm
0a9a231c5e lightning: sample once parquet file (#56205)
close pingcap/tidb#56104
2024-11-18 03:24:59 +00:00
01e2ff078e lightning: optimize a query to I_S.columns (#57117)
close pingcap/tidb#57115
2024-11-06 03:31:53 +00:00
ea55064414 lightning/importinto: set correct step to create single point allocator (#56602)
close pingcap/tidb#56476
2024-10-12 11:52:54 +00:00
c43d0589be *: replace exportloopref with copyloopvar (#56503)
close pingcap/tidb#56509
2024-10-09 12:12:43 +00:00
8f0baf4444 *: remove mock.NewContext() usage when building table meta in production code (#56348)
ref pingcap/tidb#53388
2024-09-29 13:05:25 +00:00
b427e33eb9 meta: separate reader and mutator (#56376)
ref pingcap/tidb#54436
2024-09-27 15:34:11 +00:00
2eb4dc8e03 lightning: support prepared statement and client stmt cache in logical import mode (#55482)
close pingcap/tidb#54850
2024-09-21 11:47:12 +00:00
12e5b31731 lighting: replace lightning inner contexts with manged ones (#56038)
ref pingcap/tidb#53388
2024-09-17 07:20:03 +00:00
3176b9c263 *: move XXInfo/Job, etc, out from model pkg in parser to meta (#55754)
ref pingcap/tidb#54436
2024-09-04 04:49:45 +00:00
ac9916a21e lightning: Change how the ID of taskCfg.TaskID is generated (#55505)
close pingcap/tidb#55384
2024-08-26 11:54:16 +00:00
1f095a3f47 lighting: expose limited fields for kv.Session (#55517)
ref pingcap/tidb#53388
2024-08-20 14:34:42 +00:00
adc7597a35 *: replace maps.Copy which is for map clone with std maps.Clone (#55530)
close pingcap/tidb#55527
2024-08-20 11:19:49 +00:00
a9a5b97e35 lightning: optimize checkpoint size (#55230)
close pingcap/tidb#55054
2024-08-08 08:59:10 +00:00
da7ed5cdc9 ddl: remove unused code related to prev fast create impl (#55116)
ref pingcap/tidb#54436
2024-08-01 08:13:21 +00:00