Commit Graph

76 Commits

Author SHA1 Message Date
eaf12833ea importinto: fix some check and external-id not set after SEM v2 introduced (#64170)
ref pingcap/tidb#61702
2025-10-30 00:48:58 +00:00
3dd656d7da importer: parallel init data source files (#63621)
close pingcap/tidb#62633
2025-10-15 10:18:47 +00:00
a4fda02ecb importinto: disable some options when sem enabled (#63865)
ref pingcap/tidb#61702
2025-10-09 11:23:36 +00:00
d6ca10af4f importinto: sample files to get index size ratio in nextgen (#63695)
ref pingcap/tidb#61702
2025-09-24 09:00:37 +00:00
276ed0cf95 dxf: add api to manually tune the calculation of resource related params (#63517)
ref pingcap/tidb#61702
2025-09-18 15:09:53 +00:00
941ba546bd importinto: avoid cal resource params twice and round those params (#63452)
ref pingcap/tidb#61702
2025-09-11 08:19:15 +00:00
37f8a14657 importinto: fix zero update time during validation phase and refine logs (#63425)
ref pingcap/tidb#61702
2025-09-09 12:07:54 +00:00
9825dfebc9 importinto: calculate distsql scan concurency in nextgen (#63341)
ref pingcap/tidb#61702
2025-09-03 11:52:06 +00:00
8d1a00e351 importinto: update thread count after initialized data files (#63200)
ref pingcap/tidb#61702
2025-08-28 10:38:58 +00:00
aa875ea472 dxf: calculate max_node_count and concurrency for next-gen automatically (#62995)
ref pingcap/tidb#61702
2025-08-22 14:53:38 +00:00
658fa1a43e crossks: add real-tikv tests for info schema sync (#63008)
ref pingcap/tidb#61702
2025-08-19 17:29:50 +00:00
259852d340 *: add the new SEMv2 config and implement most of the functions (#61634)
close pingcap/tidb#61582, close pingcap/tidb#61583, close pingcap/tidb#61584, close pingcap/tidb#61585
2025-08-18 13:38:47 +00:00
01bfccb1e8 importinto: add GROUP_KEY option to import jobs (#62369)
ref pingcap/tidb#61088
2025-08-15 03:22:51 +00:00
16ec21bcaa importinto: redact option cloud_storage_uri in show process list and log (#62967)
close pingcap/tidb#62966
2025-08-14 06:06:30 +00:00
8060477c03 importinto: reduce def engine size to 100G and always use max node count to adjust subtask count (#62527)
ref pingcap/tidb#61702
2025-07-21 07:41:31 +00:00
e2a9f04ee7 disttask, importer: show more detailed progress in SHOW IMPORT JOBS (#62283)
ref pingcap/tidb#61088
2025-07-18 07:59:39 +00:00
0a704a4bb7 *: collect subtask summary for IMPORT INTO tasks (#61600)
ref pingcap/tidb#61088
2025-07-04 15:12:30 +00:00
b8a0d7f075 importinto: encode and ingest step use task keyspace in nextgen (#62138)
ref pingcap/tidb#61702
2025-07-03 17:46:45 +00:00
8123a22d38 importinto: some prepare work for conflict resolve (#61426)
ref pingcap/tidb#60621
2025-06-10 12:21:59 +00:00
292c2c5caf global sort: use cluster id in file path (#61502)
close pingcap/tidb#61537
2025-06-10 06:34:40 +00:00
e315cff650 importinto: disable some feature for nextgen when sem enabled (#61370)
ref pingcap/tidb#60418
2025-05-29 06:17:55 +00:00
3ab8c7fe85 importinto: forbid set external-id explicitly and set it to keyspace name when sem enabled on nextgen (#61315)
ref pingcap/tidb#60418
2025-05-26 08:14:19 +00:00
fe769f5154 dxf: include context info in the stored error (#61289)
close pingcap/tidb#60433
2025-05-23 13:28:13 +00:00
83edc07212 import into: automatic file type recognition based on file name characteristics (#59543)
close pingcap/tidb#59540
2025-05-23 06:39:29 +00:00
0200e5d634 localbackend: merge feature branch next-gen (#60879)
ref pingcap/tidb#60418
2025-04-28 05:18:58 +00:00
08d20b8ee2 IMPORTINTO: close store inside LoadDataController (#60633)
close pingcap/tidb#60352
2025-04-18 11:18:52 +00:00
cc8d9cbbd4 lignthing/importinto: parallel reading files infos from data store (#59382)
close pingcap/tidb#56104, close pingcap/tidb#60224
2025-04-11 07:56:42 +00:00
813b32f4df import: add a hidden option to enable manual recover for debug purpose (#59674)
ref pingcap/tidb#59640
2025-04-08 08:46:48 +00:00
fe5a5175a1 importinto: disable switching to 'import mode' when global sort (#60363)
close pingcap/tidb#60361
2025-04-02 11:00:50 +00:00
d51e00e5bb globalsort: reduce number of SST ingested into TiKV (#59870) (#60045)
ref pingcap/tidb#51734, close pingcap/tidb#59962
2025-03-24 12:22:43 +00:00
e2d5d1c017 import-into: support running with 1 thread when using global sort (#58572)
close pingcap/tidb#58680
2025-02-24 06:30:23 +00:00
9099090462 dxf: add tidb_max_dist_task_nodes to specify max node count (#58937)
close pingcap/tidb#58944
2025-02-18 05:12:25 +00:00
81995d6892 csv-parser: rename fields to better reflect its meaning (#59360)
ref pingcap/tidb#44893, ref pingcap/tidb#57275
2025-02-11 05:25:19 +00:00
cf4d252af8 *: split definition out of variable pkg and refactor some reorg methods (#59146)
ref pingcap/tidb#57229, ref pingcap/tidb#57497
2025-01-23 09:24:44 +00:00
bf939fa79a *: don't handle live updates of column size (#58596)
close pingcap/tidb#58595
2025-01-06 19:42:49 +00:00
65d740f5f8 planner: Display truncate vector in EXPLAIN (#55934)
ref pingcap/tidb#54245
2024-09-29 11:51:19 +00:00
68fdf55d94 *: rename some packages of context (#56108)
close pingcap/tidb#56107
2024-09-19 07:11:03 +00:00
76098f5e16 importinto: check column assignment expressions in import into ... (#55968)
close pingcap/tidb#55497
2024-09-10 17:40:45 +00:00
3176b9c263 *: move XXInfo/Job, etc, out from model pkg in parser to meta (#55754)
ref pingcap/tidb#54436
2024-09-04 04:49:45 +00:00
dd64d25266 lightining,expression: support user variable for BuildSimpleExpr and remove PlanContext dependency in lightning context (#55617)
ref pingcap/tidb#53388
2024-09-02 04:55:18 +00:00
81cc3455c7 importinto: return error for reading user variables and using some functions (#55744)
ref pingcap/tidb#55497
2024-08-30 04:12:01 +00:00
1f095a3f47 lighting: expose limited fields for kv.Session (#55517)
ref pingcap/tidb#53388
2024-08-20 14:34:42 +00:00
f5ac1c4a45 *: support tidb_redact_log for explain (#54553)
close pingcap/tidb#54565
2024-07-23 08:45:04 +00:00
4fe31ed95e table: make slices in ColSizeDeltaBuffer private (#54539)
close pingcap/tidb#54392
2024-07-12 03:36:06 +00:00
5909899601 expression,*: remove the session context inside ParamMarker (#53534)
close pingcap/tidb#53533
2024-07-03 07:44:58 +00:00
5e6df4eeb1 expression, *: Eval param get type with context (#53656)
close pingcap/tidb#53653
2024-06-03 09:39:01 +00:00
9164182d0b config: must set line terminator when use strict-format (#53444)
close pingcap/tidb#37338
2024-05-27 08:41:50 +00:00
d626b6adf0 disttask: fix scope can't set different value (#52627)
close pingcap/tidb#52441
2024-04-26 07:19:42 +00:00
3368ccdef1 expression: introduce StaticEvalContext to provide context when evaluating expressions (#52631)
close pingcap/tidb#52630
2024-04-23 06:52:40 +00:00
482ce59bf6 expression: separate BuildContext and EvalContext (#52311)
close pingcap/tidb#52310
2024-04-03 09:42:24 +00:00