27 Commits

Author SHA1 Message Date
e60489c452 importer: sample a portion of compressed files to speed up import spec generation (#64769)
close pingcap/tidb#64770
2025-12-11 10:31:17 +00:00
779e298772 importinto/lightning: change library for parquet import (#63979)
ref pingcap/tidb#62906
2025-11-06 04:35:26 +00:00
f76844ee28 test: fix some failed tests in nextgen (#63357)
ref pingcap/tidb#61702
2025-09-03 10:03:42 +00:00
8c88b61875 tests: fix unit tests for next-gen (#62895)
ref pingcap/tidb#61702
2025-08-20 19:51:02 +00:00
8060477c03 importinto: reduce def engine size to 100G and always use max node count to adjust subtask count (#62527)
ref pingcap/tidb#61702
2025-07-21 07:41:31 +00:00
a712249d45 planner: move physical selection related logic into physical op (#62339)
ref pingcap/tidb#52714
2025-07-10 09:55:21 +00:00
b8a0d7f075 importinto: encode and ingest step use task keyspace in nextgen (#62138)
ref pingcap/tidb#61702
2025-07-03 17:46:45 +00:00
292c2c5caf global sort: use cluster id in file path (#61502)
close pingcap/tidb#61537
2025-06-10 06:34:40 +00:00
fe769f5154 dxf: include context info in the stored error (#61289)
close pingcap/tidb#60433
2025-05-23 13:28:13 +00:00
83edc07212 import into: automatic file type recognition based on file name characteristics (#59543)
close pingcap/tidb#59540
2025-05-23 06:39:29 +00:00
34afcbb9d3 executor: gopls modernize (#60787)
close pingcap/tidb#60786
2025-05-07 14:32:26 +00:00
cc8d9cbbd4 lignthing/importinto: parallel reading files infos from data store (#59382)
close pingcap/tidb#56104, close pingcap/tidb#60224
2025-04-11 07:56:42 +00:00
fe5a5175a1 importinto: disable switching to 'import mode' when global sort (#60363)
close pingcap/tidb#60361
2025-04-02 11:00:50 +00:00
d51e00e5bb globalsort: reduce number of SST ingested into TiKV (#59870) (#60045)
ref pingcap/tidb#51734, close pingcap/tidb#59962
2025-03-24 12:22:43 +00:00
cf4d252af8 *: split definition out of variable pkg and refactor some reorg methods (#59146)
ref pingcap/tidb#57229, ref pingcap/tidb#57497
2025-01-23 09:24:44 +00:00
76abd8b836 disttask,ddl: use subtask concurrency (#52286)
ref pingcap/tidb#49008
2024-04-02 04:24:16 +00:00
f2ea18c07f lightning: move lightning directory out of br (#52247)
ref pingcap/tidb#52182
2024-04-01 07:13:46 +00:00
69ab2a4de6 import into: enable [] glob matching (#51643)
ref pingcap/tidb#50752
2024-03-12 04:02:09 +00:00
fdcbd896e2 importinto: add disable_precheck option & change default threadcnt when import from select (#51077)
ref pingcap/tidb#49883
2024-02-08 13:20:46 +00:00
526d738207 expression: refactor functions to build expressions (#50997)
close pingcap/tidb#50996
2024-02-07 03:05:45 +00:00
12d273e00f importinto: executor part of import from select (#50341)
ref pingcap/tidb#49883
2024-01-17 15:52:47 +00:00
e15c517afa import/index: adjust concurrency using cpu-count of managed node depends on use case (#50091)
ref pingcap/tidb#49008
2024-01-10 08:31:54 +00:00
a7260ff598 disttask: impl slot manager in dispatcher, and check resource before start dispatcher (#49195)
close pingcap/tidb#49100
2023-12-11 13:49:48 +00:00
d99c5a547b *: fix TestSupportedSuffixForServerDisk when in RBE (#48249)
close pingcap/tidb#48225
2023-11-03 02:06:39 +00:00
1ba524c756 importinto: fix import compressed server file (#48173)
close pingcap/tidb#48049
2023-11-01 12:33:43 +00:00
99a4f35146 importinto: optimize write ingest step and fix oom (#48047)
ref pingcap/tidb#47572, close pingcap/tidb#47643
2023-10-31 11:48:38 +00:00
318e82bbb7 *: move most of the packages to 'pkg' (#47123)
ref pingcap/tidb#41241
2023-10-13 14:36:26 +00:00