Commit Graph

70 Commits

Author SHA1 Message Date
3036b6d225 importinto: support split large csv file (#46480)
ref pingcap/tidb#42930
2023-09-01 12:06:43 +00:00
d434bd9f1d import into: fix id base (#46315)
close pingcap/tidb#46174
2023-08-29 09:50:09 +00:00
a06bcc60e1 br/lightning: add DecompressConfig to external storage (#46430)
ref pingcap/tidb#42930
2023-08-29 06:18:38 +00:00
5ecf9801ae lightning: local backend use external engine part 1 (#46406)
ref pingcap/tidb#45719
2023-08-28 06:42:36 +00:00
bfacaf8887 importinto: remove useless code and add test (#46413)
ref pingcap/tidb#42930
2023-08-26 01:12:35 +00:00
8b30d3def6 *: use std/slices to replace exp/slices (#46422)
ref pingcap/tidb#45933
2023-08-25 10:27:35 +00:00
9c213aac21 lightning: fix pd http request using old address (#45680)
close pingcap/tidb#43436
2023-08-01 11:22:08 +00:00
cede736459 lightning: specify key range when switch mode (#45482) 2023-07-31 08:32:37 +00:00
f3ea1c1e06 import into: enlarge subtask size to reduce range overlap (#45488)
close pingcap/tidb#44975, ref pingcap/tidb#45005
2023-07-27 12:18:06 +00:00
d677a0c139 executor: enable revive (#45138)
ref pingcap/tidb#40786
2023-07-05 03:03:44 +00:00
dd911e70f6 lightning, importinto: reduce disk quota interval (#44927)
close pingcap/tidb#44926
2023-06-27 11:11:47 +00:00
89bf743227 importinto/lightning: do remote checksum via sql (#44803)
close pingcap/tidb#41941, ref pingcap/tidb#44711
2023-06-21 13:41:42 +00:00
8b1006f1af import into: exit ImportMode after import step (#44750)
close pingcap/tidb#44755
2023-06-19 01:53:10 +00:00
5bd56cb5d5 import into: change max pool size to num-cpu (#44749)
ref pingcap/tidb#42930
2023-06-18 06:05:10 +00:00
e606b782b1 import into: add precheck when total file size is 0 & enhance how we check task finish (#44731)
close pingcap/tidb#44729
2023-06-16 10:05:17 +00:00
d98cd4e910 import into: redact task meta after import & cancel on kill (#44712)
ref pingcap/tidb#42930
2023-06-16 07:57:11 +00:00
d996b1ab52 show import: fix show import might read data from prev line (#44724)
ref pingcap/tidb#42930
2023-06-16 07:17:18 +00:00
04eede11ec import into: add disable_tikv_import_mode option (#44587)
ref pingcap/tidb#42930
2023-06-16 07:17:11 +00:00
1b2c2725cf import into: remove sort dir because we do not support checkpoint now (#44630)
ref pingcap/tidb#42930
2023-06-16 02:27:09 +00:00
beccd05f12 import into: redact sensitive info & check active job before create (#44674)
ref pingcap/tidb#42930
2023-06-15 11:21:09 +00:00
62c93cff1b import into: make post process a step to be able cancel (#44616)
ref pingcap/tidb#42930
2023-06-15 03:59:09 +00:00
14ca3ce044 import into: add analyze table for import into (#44329)
ref pingcap/tidb#42930
2023-06-14 09:43:08 +00:00
ce174c65f2 Import into: add job management statement (#44453)
ref pingcap/tidb#42930
2023-06-12 13:47:07 +00:00
c7f2a2317d linter: enable early return (#44564)
close pingcap/tidb#40786
2023-06-12 05:43:05 +00:00
336b893423 import into: add disk quota (#44307)
ref pingcap/tidb#42930
2023-06-09 07:58:47 +00:00
20a442fead import into: precheck and register to pd (#44313)
ref pingcap/tidb#42930
2023-06-07 12:19:42 +00:00
bc45bdf3c0 lightning: deprecate send-kv-pairs (#44463)
ref pingcap/tidb#43853
2023-06-07 05:19:41 +00:00
aa99eb3848 import into: support create index by SQL (#44198)
ref pingcap/tidb#42930
2023-06-06 06:17:41 +00:00
be8d865044 import into: impl import from server disk (#44273)
ref pingcap/tidb#42930
2023-06-05 13:21:40 +00:00
ce9c0b9b8f *: refine log for GenRLimit (#44318)
close pingcap/tidb#44319
2023-06-01 05:32:42 +00:00
440e3ef117 import into: part2, impl options/formats (#44204)
ref pingcap/tidb#42930
2023-05-29 05:00:40 +00:00
5cc1c3b39e import into: switch tikv mode during import (#44067)
ref pingcap/tidb#42930
2023-05-26 07:55:39 +00:00
cd834eec65 import into: impl IMPORT INTO statement part 1 (#44122)
ref pingcap/tidb#42930
2023-05-25 05:45:38 +00:00
6e11eefcfc loaddata: don't split transaction and revert some change (#44136)
ref pingcap/tidb#44078
2023-05-25 03:43:38 +00:00
0ac7473ec9 loaddata: revert concurrent writing to TiKV and WITH detached (#44056)
ref pingcap/tidb#44078
2023-05-23 06:43:38 +00:00
6e3d52e0c5 load data: use dist framework for single-node import (#43880)
ref pingcap/tidb#42930
2023-05-22 10:31:38 +00:00
4fc5aab31d disttask: add StatementContext result for load data (#43869)
close pingcap/tidb#43868
2023-05-22 07:07:37 +00:00
ca629447df lightning: add send-kv-size to avoid oom when each kv is large on default config (#43870)
close pingcap/tidb#43853
2023-05-18 09:49:36 +00:00
6e5604f08c lightning: improve region actions for huge region number (#43807)
close pingcap/tidb#43510
2023-05-17 10:01:36 +00:00
f1c388eae5 disttask: add checksum verification for load data (#43716)
close pingcap/tidb#43715
2023-05-17 09:13:49 +00:00
6043234d98 lightning: add option to control pause scheduler scope (#43762)
close pingcap/tidb#43657
2023-05-15 06:19:17 +00:00
56dacd0265 lightning: fix oom when mem/cpu ratio is low (#43729)
close pingcap/tidb#43728
2023-05-12 03:36:21 +00:00
5f11392afc load data: sync rewrite set expr for concurrent load (#43492)
ref pingcap/tidb#42930
2023-05-09 08:46:58 +00:00
e2277cf7dd load data: time and log for different phase, use err group in chunk_process (#43245)
ref pingcap/tidb#42930
2023-04-26 07:29:52 +00:00
78e614f0e6 loaddata: make some test use 1 thread explicitly (#43394)
ref pingcap/tidb#42930
2023-04-26 02:29:51 +00:00
f224d95093 lightning: reuse worker-concurr and remove range-concurr in local backend (#43348)
ref pingcap/tidb#42930
2023-04-25 07:13:52 +00:00
1c49453136 disttask: enable load data for distribute framework (#43107)
close pingcap/tidb#43106
2023-04-24 11:04:46 +00:00
66c36ededb *: support load data for table with generated columns (#43108)
close pingcap/tidb#39885
2023-04-20 12:27:19 +08:00
2aa72b7db6 load data: physical mode part 4 (#43162)
ref pingcap/tidb#42930
2023-04-19 19:01:20 +08:00
d33b9ae173 load data: check table empty or not for physical mode (#43157)
ref pingcap/tidb#42930
2023-04-19 13:21:19 +08:00