ab513bf0ca
ddl: support concurrent ddl ( #32169 )
...
* 1. init ddl tables
create tidb_ddl_job, tidb_ddl_reorg, tidb_ddl_history tables with raw meta write, these 3 tables is use to replace the ddl job queue and reorg and history hash table
* 2. setup concurrent ddl env and add ddl worker pool
adds the ddl worker pool definition, the ddl job manager will find a job and ship it to a worker in the worker pool.
Also, this commit provides a sessionctx wrapper, only use in ddl relate. it just wraps begin, commit and execute
* 3. add ddl manager to handle ddl job
* 4. reorg handler for concurrent ddl
just implements the partner of the reorg information.
* 5. manage ddl jobs for concurrent ddl
add the partner of add job, delete job and many others related to history job
because many of the functions need a session now, we just change the caller
* 6. add metrics for concurrent ddl
add metrics
* 7. support multiple tables
* 8. fix test
* 9. migrate ddl between table and queue
support switch between the old and new ddl framework, migrate the existing ddl job between queue and table
* 10. check tikv version and set reorg worker count according cpu count
* *: add featuretag on tests
Signed-off-by: Weizhen Wang <wangweizhen@pingcap.com >
* use a determined table id for 3 tables
* remove ctx value
* add GetSchemaVersionWithNonEmptyDiff function
* address tangenta and zimulala comment
* use only one etcd path
* make ActionRenameTable support multi-schema
* reset sql digest to make top sql work correct
* add comment
* fix test
* remove 0 for schema version lock
Co-authored-by: xiongjiwei <xiongjiwei1996@outlook.com >
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io >
Co-authored-by: wjHuang <huangwenjun1997@gmail.com >
2022-07-20 21:20:07 +08:00
fbbae462f1
planner: expose two variables related to the cost model ( #36356 )
...
ref pingcap/tidb#35240
2022-07-20 20:45:09 +08:00
fd6f5dd798
*: GC-aware memory track for analyze ( #35248 )
...
close pingcap/tidb#34996
2022-07-20 20:17:09 +08:00
8a91606f5b
planner: fix bug that skew distinct agg rule failed to generate plan ( #36357 )
...
close pingcap/tidb#36350
2022-07-20 19:47:08 +08:00
b54ea725ba
planner: add more test cases for outer join's leading hint ( #36125 )
...
ref pingcap/tidb#29932
2022-07-20 19:11:09 +08:00
a52656fde4
*: add binary plan in stmt summary and slow log/query ( #35698 )
...
close pingcap/tidb#35889
2022-07-20 18:21:09 +08:00
6256c347a7
parser: let LONG BYTE compatible with MySQL ( #36314 )
...
close pingcap/tidb#36239
2022-07-20 17:29:09 +08:00
a6b3f3d454
compact: Add syntax for COMPACT command without specifying the replica ( #36368 )
...
ref pingcap/tidb#34749
2022-07-20 16:53:09 +08:00
d00b984f4c
executor, statistics: analyze use MaxUint64 ts to read data ( #35232 )
...
close pingcap/tidb#35233
2022-07-20 16:09:08 +08:00
a5ec60fceb
expression: fix accuracy loss problem in function COALESCE ( #36264 )
...
close pingcap/tidb#31600
2022-07-20 15:35:08 +08:00
dff77a9394
*: remove repetitive linter ( #36334 )
2022-07-20 14:31:08 +08:00
4b50e67452
statistics: separte complex file and forbid embedding ( #36318 )
...
close pingcap/tidb#36284
2022-07-20 14:03:08 +08:00
e37fb3e13d
docs: RFC of Session Manager ( #36273 )
...
close pingcap/tidb#36272
2022-07-20 13:35:07 +08:00
98c27d86b9
ddl: fix data race in the setDDLLabelForDiagnosis ( #36207 )
...
close pingcap/tidb#36343
2022-07-20 13:09:08 +08:00
c087f6dc2a
table partition: update job dependence for exchange partition with table ( #35751 )
...
close pingcap/tidb#35842
2022-07-20 12:41:08 +08:00
460d57c8ba
log-backup: fix error that check whether partitionInfo.Columns is nil while it is [] ( #36349 )
...
close pingcap/tidb#36348
2022-07-20 12:15:08 +08:00
0cecfc85af
util/paging: choose min paging size default value as 128, and max value as 8192 ( #36331 )
...
close pingcap/tidb#36328
2022-07-19 21:57:08 +08:00
a0cced2167
executor: fix show create table with set and enum ( #36327 )
...
close pingcap/tidb#36317
2022-07-19 20:07:08 +08:00
a1abdbe38a
br: more i t for restore priv table ( #36330 )
...
ref pingcap/tidb#35395
2022-07-19 19:37:08 +08:00
51b8884fea
expression: fix the issue that extracting day_microsecond/day_second/day_minute/day_hour from Time type emits wrong result ( #36297 )
...
close pingcap/tidb#34998
2022-07-19 19:07:07 +08:00
e0b4835335
resolve-lock: resolve legacy lock if safepoint < lastSafepoint and skip runGCJob() ( #36300 )
...
ref pingcap/tidb#35428
2022-07-19 18:39:07 +08:00
afd71bdefd
executor: support dispatch cop request to closest replica adaptively ( #35927 )
...
ref pingcap/tidb#35926
2022-07-19 17:59:07 +08:00
d48f40a09c
util, session: enable paging for more ut ( #36259 )
...
ref pingcap/tidb#35633
2022-07-19 17:33:08 +08:00
f84554bd07
expression: fix binary str numeric result type ( #36258 )
...
close pingcap/tidb#27698
2022-07-19 16:43:07 +08:00
597b2cc9cf
doc: remove the imcomplete design document ( #36316 )
...
close pingcap/tidb#32031
2022-07-19 16:05:07 +08:00
11625d46af
domain: fix unstable test ( #36308 )
...
close pingcap/tidb#36310
2022-07-19 15:25:07 +08:00
486ee8ecd0
statistics: fix panic when cmsketch is dropped ( #36299 )
...
ref pingcap/tidb#36284
2022-07-19 14:57:07 +08:00
a57dd37640
planner: add a hint for CTE ( #34574 )
...
close pingcap/tidb#17472
2022-07-19 14:31:07 +08:00
0026324ac6
*: enable revive for more components ( #36296 )
2022-07-19 12:47:07 +08:00
d90cee5002
planner: push down top selection to TiFlash ( #36256 )
...
close pingcap/tidb#34192
2022-07-19 09:57:07 +08:00
de017e9eea
expression: forbid aggregate function with json type pushdown to tiflash wrongly ( #36271 )
...
close pingcap/tidb#28753
2022-07-19 01:17:07 +08:00
8af8341cca
planner: Table scan in TiFlash not keep order ( #36232 )
...
close pingcap/tidb#36231
2022-07-19 00:51:07 +08:00
f364b90010
telemetry: add telemetry for fast mode ( #36222 )
...
ref pingcap/tidb#35850
2022-07-19 00:23:07 +08:00
c8e27fe0b9
ddl: handle rolling back modify columns correctly ( #36282 )
...
ref pingcap/tidb#14766
2022-07-18 23:55:07 +08:00
63f5d286ff
log-backup: add version check for pitr ( #36268 )
...
close pingcap/tidb#36267
2022-07-18 23:27:07 +08:00
39f8eaa997
br: add search key debug command ( #36274 )
...
* feat: add search key debug command
* feat: modify to merge default cf and write cf keys
* feat: modify to read backup meta files concurrently
* feat: add unit test and comments
* fix: check error returned from task.GetStorage
* feat: resolve reviewing problems
* fix: modify removed function
Co-authored-by: Zak Zhao <57036248+joccau@users.noreply.github.com >
Co-authored-by: Weizhen Wang <wangweizhen@pingcap.com >
Co-authored-by: 3pointer <luancheng@pingcap.com >
2022-07-18 22:47:36 +08:00
98595124b2
log-backup: added more metrics and hint; fixed a bug may cause inf loop ( #36228 )
...
ref pingcap/tidb#29501
2022-07-18 22:17:07 +08:00
8b30e52bcf
ddl: only use one schema version for the non-revertible step ( #36262 )
...
ref pingcap/tidb#14766
2022-07-18 21:49:07 +08:00
67be460efe
planner: enable the new cost interface by default ( #36241 )
...
ref pingcap/tidb#35240
2022-07-18 21:21:08 +08:00
48830de368
*: update client-go to improve performance for sortRegion ( #36275 )
2022-07-18 17:49:08 +08:00
4cade24ed6
*: support the binary plan through EXPLAIN format='binary' and tidb_decode_binary_plan ( #36221 )
...
ref pingcap/tidb#35889
2022-07-18 16:27:07 +08:00
5dcf2ef6ed
telemetry: add telemetry for tidb_enable_paging ( #36261 )
...
close pingcap/tidb#36260
2022-07-18 15:59:07 +08:00
c28bac3d72
planner: support rewrite for skew group distinct aggregate to avoid skew execution ( #36181 )
...
close pingcap/tidb#36169
2022-07-18 15:29:07 +08:00
df31ac65a4
execdetails: output write details in RuntimeStats ( #35814 )
...
ref pingcap/tidb#34487
2022-07-18 15:01:07 +08:00
1acc178da5
planner: not use DescScanFactor when rows is less than smallScanThreshold for compatibility ( #36263 )
2022-07-18 14:36:09 +08:00
0d6741039a
log-backup: calculate global checkpoint-ts from storage ( #36255 )
...
close pingcap/tidb#36246
2022-07-18 14:09:07 +08:00
1883a4444d
br: restore privilege tables on full restore by default ( #35397 )
...
close pingcap/tidb#35395
2022-07-18 13:11:06 +08:00
3c2a5cff5c
bazel: remove timeout ( #36257 )
2022-07-18 12:43:07 +08:00
8c380e3107
ddl: support alter table options for multi-schema change ( #36236 )
...
ref pingcap/tidb#14766 , close pingcap/tidb#36234
2022-07-18 10:39:07 +08:00
f675cf619b
ddl: ban set tiflash mode under unsupported case ( #36211 )
...
ref pingcap/tidb#35850
2022-07-18 09:53:06 +08:00
