From 60c002ba91e352fd82a777f720aba5c64ad07c42 Mon Sep 17 00:00:00 2001 From: Calvin Kirs Date: Mon, 26 Feb 2024 12:06:56 +0800 Subject: [PATCH] [Docs](Job)Add job Docs (#28186) * [Docs](Job)Add job Docs * format * Fix job error --- .../sql-functions/table-functions/job.md | 96 ++++++++++++++++++ .../sql-functions/table-functions/tasks.md | 69 +++++++++++-- .../Alter/PAUSE-JOB.md | 14 +-- .../Alter/RESUME-JOB.md | 12 ++- .../Create/CREATE-JOB.md | 76 +++++++++++---- .../{Alter/STOP-JOB.md => Drop/DROP-JOB.md} | 20 ++-- .../Show-Statements/SHOW-JOB-TASK.md | 74 -------------- .../sql-reference/Show-Statements/SHOW-JOB.md | 93 ------------------ .../sql-functions/table-functions/job.md | 97 +++++++++++++++++++ .../sql-functions/table-functions/tasks.md | 65 ++++++++++++- .../Alter/PAUSE-JOB.md | 10 +- .../Alter/RESUME-JOB.md | 10 +- .../Create/CREATE-JOB.md | 48 +++++---- .../{Alter/STOP-JOB.md => Drop/DROP-JOB.md} | 14 +-- .../Show-Statements/SHOW-JOB-TASK.md | 75 -------------- .../sql-reference/Show-Statements/SHOW-JOB.md | 93 ------------------ 16 files changed, 442 insertions(+), 424 deletions(-) create mode 100644 docs/en/docs/sql-manual/sql-functions/table-functions/job.md rename docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/{Alter/STOP-JOB.md => Drop/DROP-JOB.md} (79%) delete mode 100644 docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB-TASK.md delete mode 100644 docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/table-functions/job.md rename docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/{Alter/STOP-JOB.md => Drop/DROP-JOB.md} (80%) delete mode 100644 docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB-TASK.md delete mode 100644 docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md diff --git a/docs/en/docs/sql-manual/sql-functions/table-functions/job.md b/docs/en/docs/sql-manual/sql-functions/table-functions/job.md new file mode 100644 index 0000000000..372c63f809 --- /dev/null +++ b/docs/en/docs/sql-manual/sql-functions/table-functions/job.md @@ -0,0 +1,96 @@ +--- +{ + "title": "JOB", + "language": "en" +} +--- + + + +## `job` + +### Name + + + +job + + + +### description + +Table function, generates a temporary table of jobs, which allows you to view the information of jobs in the current Doris cluster. + +This function is used in the FROM clause. + +#### syntax + +**parameter description** + +| parameter | description | type | required | +|:----------|:------------|:-------|:---------| +| type | job type | string | yes | + +the **type** supported types +- insert: insert into type job + + +##### Insert Job + +The table schema of `tasks("type"="insert");` tvf： + +``` +mysql> desc function jobs("type"="insert") ++-------------------+------+------+-------+---------+-------+ +| Field | Type | Null | Key | Default | Extra | ++-------------------+------+------+-------+---------+-------+ +| Id | TEXT | No | false | NULL | NONE | +| Name | TEXT | No | false | NULL | NONE | +| Definer | TEXT | No | false | NULL | NONE | +| ExecuteType | TEXT | No | false | NULL | NONE | +| RecurringStrategy | TEXT | No | false | NULL | NONE | +| Status | TEXT | No | false | NULL | NONE | +| ExecuteSql | TEXT | No | false | NULL | NONE | +| CreateTime | TEXT | No | false | NULL | NONE | +| Comment | TEXT | No | false | NULL | NONE | ++-------------------+------+------+-------+---------+-------+ +``` + +### example + +``` +mysql> select * from jobs("type"="insert") where Name='kris'\G +*************************** 1. row *************************** + Id: 10069 + Name: kris + Definer: root + ExecuteType: RECURRING +RecurringStrategy: EVERY 3 SECOND STARTS 2023-12-06 14:44:47 + Status: RUNNING + ExecuteSql: insert into address select * from mysqluser.orders.address where 'create_time' >= days_add(now(),-1) + CreateTime: 2023-12-06 14:44:44 + Comment: load mysql address datas +1 row in set (0.04 sec) +``` + +### keywords + + job, insert, schedule + diff --git a/docs/en/docs/sql-manual/sql-functions/table-functions/tasks.md b/docs/en/docs/sql-manual/sql-functions/table-functions/tasks.md index e924cc8d6d..4cf497c891 100644 --- a/docs/en/docs/sql-manual/sql-functions/table-functions/tasks.md +++ b/docs/en/docs/sql-manual/sql-functions/table-functions/tasks.md @@ -28,19 +28,56 @@ under the License. ### Name + + tasks + + ### description -Table function, generating a temporary task table, which can view task information in a certain task type. +Table function, generates a temporary table of tasks, which allows you to view the information of tasks generated by jobs in the current Doris cluster. -This function is used in the from clause. +This function is used in the FROM clause. #### syntax -`tasks("type"="")` +`tasks("type"="insert");` +**parameter description** -tasks("type"="mv") Table structure: +| parameter | description | type | required | +|:----------|:------------|:-------|:---------| +| type | job type | string | yes | + +the **type** supported types +- insert: insert into type job +- mv: materilized view type job + +##### Insert tasks + +The table schema of `tasks("type"="insert");` tvf： + +``` +mysql> desc function tasks("type"="insert"); ++---------------+------+------+-------+---------+-------+ +| Field | Type | Null | Key | Default | Extra | ++---------------+------+------+-------+---------+-------+ +| TaskId | TEXT | No | false | NULL | NONE | +| JobId | TEXT | No | false | NULL | NONE | +| Label | TEXT | No | false | NULL | NONE | +| Status | TEXT | No | false | NULL | NONE | +| EtlInfo | TEXT | No | false | NULL | NONE | +| TaskInfo | TEXT | No | false | NULL | NONE | +| ErrorMsg | TEXT | No | false | NULL | NONE | +| CreateTimeMs | TEXT | No | false | NULL | NONE | +| FinishTimeMs | TEXT | No | false | NULL | NONE | +| TrackingUrl | TEXT | No | false | NULL | NONE | +| LoadStatistic | TEXT | No | false | NULL | NONE | +| User | TEXT | No | false | NULL | NONE | ++---------------+------+------+-------+---------+-------+ +12 rows in set (0.01 sec) +``` +##### MV Tasks ```sql mysql> desc function tasks("type"="mv"); +-----------------------+------+------+-------+---------+-------+ @@ -86,8 +123,27 @@ mysql> desc function tasks("type"="mv"); * NeedRefreshPartitions: The partition information that needs to be refreshed for this task * CompletedPartitions: The partition information that has been refreshed for this task * Progress: Task running progress - ### example +#### Insert Tasls +``` +mysql> select * from tasks("type"="insert") limit 1 \G +*************************** 1. row *************************** + TaskId: 667704038678903 + JobId: 10069 + Label: 10069_667704038678903 + Status: FINISHED + EtlInfo: \N + TaskInfo: cluster:N/A; timeout(s):14400; max_filter_ratio:0.0; priority:NORMAL + ErrorMsg: \N + CreateTimeMs: 2023-12-08 16:46:57 + FinishTimeMs: 2023-12-08 16:46:57 + TrackingUrl: +LoadStatistic: {"Unfinished backends":{},"ScannedRows":0,"TaskNumber":0,"LoadBytes":0,"All backends":{},"FileNumber":0,"FileSize":0} + User: root +1 row in set (0.05 sec) + +``` +#### MV Tasks 1. View tasks for all materialized views @@ -101,6 +157,7 @@ mysql> select * from tasks("type"="mv"); mysql> select * from tasks("type"="mv") where JobName="inner_mtmv_75043"; ``` + ### keywords - tasks + tasks, job, insert, mv, materilized view \ No newline at end of file diff --git a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/PAUSE-JOB.md b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/PAUSE-JOB.md index 8f8dcbce6d..5920c14398 100644 --- a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/PAUSE-JOB.md +++ b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/PAUSE-JOB.md @@ -32,22 +32,22 @@ PAUSE JOB ### Description -User suspends a JOB job. A stopped job can be resumed with RESUME JOB. +User suspends a JOB. A stopped job can be resumed with RESUME JOB. ```sql -PAUSE JOB FOR job_name; +PAUSE JOB where jobName = 'job_name'; ``` ### Example 1. Pause the job named test1. - ```sql - PAUSE JOB FOR test1; - ``` + ```sql + PAUSE JOB where jobName = 'test1'; + ``` -###Keywords +### Keywords - PAUSE, JOB + PAUSE, JOB ### Best Practice diff --git a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/RESUME-JOB.md b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/RESUME-JOB.md index 49eb11b230..69608f1416 100644 --- a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/RESUME-JOB.md +++ b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/RESUME-JOB.md @@ -35,20 +35,22 @@ RESUME JOB Used to restart a JOB in PAUSE state. The restarted job will continue to be executed periodically. A JOB in STOP state cannot be resumed. ```sql -RESUME JOB FOR job_name; +RESUME JOB WHERE jobName = 'job_name'; ``` ### Example 1. Restart the JOB named test1. - ```sql - RESUME JOB FOR test1; - ``` + ```sql + RESUME JOB WHERE jobName = 'test1'; + ``` ### Keywords - RESUME, JOB + RESUME, JOB ### Best Practice + + diff --git a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-JOB.md b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-JOB.md index f173cfee93..9407d031b8 100644 --- a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-JOB.md +++ b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-JOB.md @@ -32,15 +32,40 @@ CREATE JOB ### Description -Doris Job is a task that runs according to a predetermined plan and is used to trigger predefined actions at a specific time or at a specified time interval, thereby helping us to automate some tasks. Functionally, it is similar to the operating system's -Timing tasks (such as: cron in Linux, scheduled tasks in Windows). But Doris's job scheduling can be accurate to the second level. +Doris Job is a task that runs according to a predefined schedule, triggering predefined actions at specific times or intervals to help automate certain tasks. In terms of functionality, it is similar to scheduled tasks on operating systems (e.g., cron in Linux, scheduled tasks in Windows).↳ -There are two types of jobs: `ONE_TIME` and `BATCH`. Among them, the `ONE_TIME` type Job will be triggered at the specified time point, which is mainly used for one-time tasks, while the `BATCH` type Job will be triggered cyclically within the specified time interval. -Mainly used for tasks that are executed periodically. +There are two types of Jobs: ONE_TIME and RECURRING. The ONE_TIME type of Job triggers at a specified time point and is mainly used for one-time tasks, while the RECURRING type of Job triggers at specified time intervals and is used for periodically recurring tasks. The RECURRING type of Job can specify a start time and an end time using STARTS\ENDS. If the start time is not specified, the first execution time is set to the current time plus one scheduling period. If the end time is specified and the task completes execution by reaching the end time (or exceeds it, or the next execution period exceeds the end time), the Job status is updated to FINISHED, and no more Tasks are generated. -Currently only ***ADMIN*** permissions are supported for this operation. +#### Job Status -grammar: +A Job has four states (RUNNING, STOPPED, PAUSED, FINISHED), with the initial state being RUNNING. A Job in the RUNNING state generates Tasks based on the specified scheduling period. When a Job completes execution and reaches the end time, the status changes to FINISHED. + +A Job in the RUNNING state can be paused, which means it will no longer generate Tasks. + +A Job in the PAUSED state can be resumed by performing the RESUME operation, changing the state to RUNNING. + +A Job in the STOPPED state is triggered by the user, which cancels the running Job and then deletes it. + +A Job in the FINISHED state remains in the system for 24 hours and is deleted after that. + +#### Task status + +A Job only describes the job information, and the execution generates Tasks. The Task status can be PENDING, RUNNING, SUCCESS, FAILED, or CANCELED. + +PENDING indicates that the trigger time has been reached but resources are awaited for running. Once resources are allocated, the status changes to RUNNING. When the execution is successful or fails, the status changes to SUCCESS or FAILED, respectively. + +CANCELED indicates the cancellation status. The final status of a Task is persisted as SUCCESS or FAILED. Other statuses can be queried while the Task is running, but they become invisible after a restart. Only the latest 100 Task records are retained. + +#### Permissions + +Currently, only users with the ADMIN role can perform this operation. + +#### Related Documentation + +[PAUSE-JOB](../Alter/PAUSE-JOB.md),[RESUME-JOB](../Alter/RESUME-JOB.md),[DROP-JOB](../Drop/DROP-JOB.md), [TVF-JOB](../../../sql-functions/table-functions/job.md), +[TVF-TASKS](../../../sql-functions/table-functions/tasks) + +### Grammar ```sql CREATE @@ -64,9 +89,9 @@ interval: A valid Job statement must contain the following -- The keyword CREATE JOB plus the job name, which identifies unique events within a db. +- The keyword CREATE JOB plus the job name, which uniquely identifies the event within a database. The job name must be globally unique, and if a JOB with the same name already exists, an error will be reported. We reserve the inner_ prefix for internal use, so users cannot create names starting with ***inner_***. - The ON SCHEDULE clause, which specifies the type of Job and when and how often to trigger it. -- The DO clause, which specifies the actions that need to be performed when the Job job is triggered. +- The DO clause, which specifies the actions that need to be performed when the Job is triggered. Here is a minimal example: @@ -79,26 +104,27 @@ This statement means to create a job named my_job to be executed every minute, a The SCHEDULE statement is used to define the execution time, frequency and duration of the job, which can specify a one-time job or a periodic job. - AT timestamp - For one-time events, it specifies that the event is only executed once at a given date and time timestamp, which must contain the date and time +Format: 'YYYY-MM-DD HH:MM:SS'. Used for one-time events, it specifies that the event should only be executed once at the given date and time. Once the execution is complete, the Job status changes to FINISHED. - EVERY Indicates that the operation is repeated periodically, which specifies the execution frequency of the job. After the keyword, a time interval should be specified, which can be days, hours, minutes, seconds, and weeks. - - interval + - interval - Used to specify the job execution frequency, it can be days, hours, minutes, seconds, weeks. For example: `1 DAY` means execute once a day, `1 HOUR` means execute once an hour, `1 MINUTE` means execute once every minute, `1 WEEK` means execute once a week, `1 SECOND` means execute once every second . + Used to specify the Job execution frequency, which can be `day`, `hour`, `minute`, or `week`. For example, 1 `DAY` means the Job will run once every day, 1 `HOUR` means once every hour, 1 `MINUTE` means once every minute, and `1 The CREATE JOB statement is used to create a job in a database. A job is a task that can be scheduled to run at specific times or intervals to automate certain actions. - - STARTS timestamp + - STARTS timestamp(optional) - It is used to specify the start time of the job. If not specified, it will be executed from the next time point of the current time. + Format: 'YYYY-MM-DD HH:MM:SS'. It is used to specify the start time of the job. If not specified, the job starts executing from the next occurrence based on the current time. The start time must be greater than the current time. - - ENDS timestamp + - ENDS timestamp(optional) + + Format: 'YYYY-MM-DD HH:MM:SS'. It is used to specify the end time of the job. If not specified, it means the job executes indefinitely. The end date must be greater than the current time. If a start time (↳STARTS) is specified, the end time must be greater than the start time. - Used to specify the end time of the job, if not specified, it means permanent execution. - DO - It is used to specify the operation that needs to be performed when the job is triggered. Currently, all ***INSERT, UPDATE*** operations are supported. We will support more operations in the future. + It is used to specify the operation that needs to be performed when the job is triggered. Currently, all ***INSERT*** operations are supported. We will support more operations in the future. ### Example @@ -122,8 +148,20 @@ Create a periodic Job, which will start to execute at 2020-01-01 00:00:00, and e CREATE JOB my_job ON SCHEDULE EVERY 1 DAY STARTS '2020-01-01 00:00:00' ENDS '2020-01-01 00:10:00' DO INSERT INTO db1.tbl1 SELECT * FROM db2.tbl2 create_time >= days_add (now(),-1); ``` +### CONFIG + +#### fe.conf + +- job_dispatch_timer_job_thread_num: Number of threads used for dispatching scheduled tasks. Default value is 2. If there are a large number of periodically executed tasks, this parameter can be increased. +- job_dispatch_timer_job_queue_size: Size of the queue used for storing scheduled tasks when there is task accumulation. Default value is 1024. If there are a large number of tasks triggered at the same time, this parameter can be increased. Otherwise, the queue may become full and submitting tasks will be blocked, causing subsequent tasks to be unable to submit. +- finished_job_cleanup_threshold_time_hour: Time threshold, in hours, for cleaning up completed tasks. Default value is 24 hours. +- job_insert_task_consumer_thread_num: Number of threads used for executing Insert tasks. The value should be greater than 0, otherwise the default value is 5. + +### Best Practice + +- Properly manage Jobs to avoid triggering a large number of Jobs simultaneously, which can lead to task accumulation and affect the normal operation of the system. +- Set the execution interval of tasks within a reasonable range, ensuring that it is at least greater than the task execution time. + ### Keywords - CREATE, JOB - -### Best Practice \ No newline at end of file + CREATE, JOB, SCHEDULE \ No newline at end of file diff --git a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/STOP-JOB.md b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Drop/DROP-JOB.md similarity index 79% rename from docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/STOP-JOB.md rename to docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Drop/DROP-JOB.md index 05b9d08111..ecbc1837ee 100644 --- a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/STOP-JOB.md +++ b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Drop/DROP-JOB.md @@ -1,6 +1,6 @@ --- { -"title": "STOP-JOB", +"title": "DROP-JOB", "language": "en" } --- @@ -24,31 +24,31 @@ specific language governing permissions and limitations under the License. --> -## STOP-JOB +## DROP-JOB ### Name -STOP JOB +DROP JOB ### Description -User stops a JOB job. A stopped job cannot be rerun. +User stops a JOB. A stopped job cannot be resumed. ```sql -STOP JOB FOR job_name; +DROP JOB WHERE jobName = 'job_name'; ``` ### Example -1. Stop the job named test1. +1. DROP the job named test1. - ```sql - STOP JOB FOR test1; - ``` + ```sql + DROP JOB where jobName = 'test1'; + ``` ### Keywords - STOP, JOB + DROP, JOB ### Best Practice diff --git a/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB-TASK.md b/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB-TASK.md deleted file mode 100644 index d816059510..0000000000 --- a/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB-TASK.md +++ /dev/null @@ -1,74 +0,0 @@ ---- -{ -"title": "SHOW-JOB-TASK", -"language": "en" -} ---- - - -## SHOW-JOB-TASK - -### Name - -SHOW JOB TASK - -### Description - -This statement is used to display the list of execution results of JOB subtasks, and the latest 20 records will be kept by default. - -grammar: - -```sql -SHOW JOB TASKS FOR job_name; -``` - - - -Result description: - -``` - JobId: JobId - TaskId: TaskId - StartTime: start execution time - EndTime: end time - Status: status - Result: execution result - ErrMsg: error message -``` - -* State - - There are the following 2 states: - * SUCCESS - * FAIL - -### Example - -1. Display the task execution list of the JOB named test1 - - ```sql - SHOW JOB TASKS FOR test1; - ``` - -###Keywords - - SHOW, JOB, TASK - -### Best Practice \ No newline at end of file diff --git a/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md b/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md deleted file mode 100644 index ec11fd3168..0000000000 --- a/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -{ -"title": "SHOW-JOB", -"language": "en" -} ---- - - - -## SHOW-JOB - -###Name - -SHOW JOB - -### Description - -This statement is used to display the running status of the JOB job - -grammar: - -```sql -SHOW JOBS|JOB FOR job_name; -``` - -SHOW JOBS is used to display the running status of all jobs under the current DB, and SHOW JOB FOR job_name is used to display the running status of the specified job. - -Result description: - -``` - Id: JobId - Db: database name - Name: Job name - Definer: create user - TimeZone: time zone - ExecuteType: RECURRING means cyclic scheduling, that is, the scheduling time specified by the every statement, ONCE_TIME means a one-time task. - ExecuteAT: ONCE_TIME The execution start time of the task - ExecuteInterval: Interval of periodic scheduling tasks - ExecuteInterval: The time interval unit for periodic scheduling tasks - STARTS: The start time of periodic scheduled task settings - ENDS: The end time set by the periodic scheduling task - Status: Job status - LastExecuteFinishTime: The time when the last execution was completed - ErrorMsg: error message - Comment: Remarks - - -``` - -* State - - There are the following 5 states: - * RUNNING: running - * PAUSED: Paused - * STOPPED: end (manually triggered by the user) - * FINISHED: Finished - -### Example - -1. Display all JOBs under the current DB. - - ```sql - SHOW JOBS; - ``` - -2. Display the JOB named test1 - - ```sql - SHOW JOB FOR test1; - ``` - -###Keywords - - SHOW, JOB - -### Best Practice \ No newline at end of file diff --git a/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/job.md b/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/job.md new file mode 100644 index 0000000000..f121bd7683 --- /dev/null +++ b/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/job.md @@ -0,0 +1,97 @@ +--- +{ + "title": "JOB", + "language": "zh-CN" +} +--- + + + +## `job` + +### Name + + + +job + + + +### description + +表函数，生成 job 临时表，可以查看当前 doris 集群中的 job 信息。 + +该函数用于from子句中。 + +#### syntax + +`jobs("type"="insert");` + +**参数说明** + +| 参数名 | 说明 | 类型 | 是否必填 | +|:-----|:-----|:-------|:-----| +| type | 作业类型 | string | 是 | + +type 支持的类型： + +- insert：insert into 类型的任务。 + +##### Insert Job + +`jobs("type"="insert");`表结构： + +``` +mysql> desc function jobs("type"="insert") ++-------------------+------+------+-------+---------+-------+ +| Field | Type | Null | Key | Default | Extra | ++-------------------+------+------+-------+---------+-------+ +| Id | TEXT | No | false | NULL | NONE | +| Name | TEXT | No | false | NULL | NONE | +| Definer | TEXT | No | false | NULL | NONE | +| ExecuteType | TEXT | No | false | NULL | NONE | +| RecurringStrategy | TEXT | No | false | NULL | NONE | +| Status | TEXT | No | false | NULL | NONE | +| ExecuteSql | TEXT | No | false | NULL | NONE | +| CreateTime | TEXT | No | false | NULL | NONE | +| Comment | TEXT | No | false | NULL | NONE | ++-------------------+------+------+-------+---------+-------+ +``` + +### example + +``` +mysql> select * from jobs("type"="insert") where Name='kris'\G +*************************** 1. row *************************** + Id: 10069 + Name: kris + Definer: root + ExecuteType: RECURRING +RecurringStrategy: EVERY 3 SECOND STARTS 2023-12-06 14:44:47 + Status: RUNNING + ExecuteSql: insert into address select * from mysqluser.orders.address where 'create_time' >= days_add(now(),-1) + CreateTime: 2023-12-06 14:44:44 + Comment: load mysql address datas +1 row in set (0.04 sec) +``` + +### keywords + + job, insert \ No newline at end of file diff --git a/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/tasks.md b/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/tasks.md index 0a4ee24bbb..ff6deac441 100644 --- a/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/tasks.md +++ b/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/tasks.md @@ -28,19 +28,56 @@ under the License. ### Name + + tasks + + ### description -表函数，生成任务临时表，可以查看某个任务类型中的task信息。 +表函数，生成 tasks 临时表，可以查看当前 doris 集群中的 job 产生的 tasks 信息。 该函数用于 from 子句中。 #### syntax -`tasks("type"="")` +`tasks("type"="insert");` +**参数说明** -tasks("type"="mv")表结构： +| 参数名 | 说明 | 类型 | 是否必填 | +|:-----|:-------|:-------|:-----| +| type | 作业类型 | string | 是 | + +type 支持的类型： + +- insert：insert into 类型的任务。 +- mv: mv 类型的任务 + +##### Insert tasks +`tasks("type"="insert");`表结构： +``` +mysql> desc function tasks("type"="insert"); ++---------------+------+------+-------+---------+-------+ +| Field | Type | Null | Key | Default | Extra | ++---------------+------+------+-------+---------+-------+ +| TaskId | TEXT | No | false | NULL | NONE | +| JobId | TEXT | No | false | NULL | NONE | +| Label | TEXT | No | false | NULL | NONE | +| Status | TEXT | No | false | NULL | NONE | +| EtlInfo | TEXT | No | false | NULL | NONE | +| TaskInfo | TEXT | No | false | NULL | NONE | +| ErrorMsg | TEXT | No | false | NULL | NONE | +| CreateTimeMs | TEXT | No | false | NULL | NONE | +| FinishTimeMs | TEXT | No | false | NULL | NONE | +| TrackingUrl | TEXT | No | false | NULL | NONE | +| LoadStatistic | TEXT | No | false | NULL | NONE | +| User | TEXT | No | false | NULL | NONE | ++---------------+------+------+-------+---------+-------+ +12 rows in set (0.01 sec) +``` + +##### MV tasks ```sql mysql> desc function tasks("type"="mv"); +-----------------------+------+------+-------+---------+-------+ @@ -67,7 +104,6 @@ mysql> desc function tasks("type"="mv"); +-----------------------+------+------+-------+---------+-------+ 18 rows in set (0.00 sec) ``` - * TaskId：task id * JobId：job id * JobName：job名称 @@ -88,7 +124,26 @@ mysql> desc function tasks("type"="mv"); * Progress：task运行进度 ### example +#### Insert Tasks +``` +mysql> select * from tasks("type"="insert") limit 1 \G +*************************** 1. row *************************** + TaskId: 667704038678903 + JobId: 10069 + Label: 10069_667704038678903 + Status: FINISHED + EtlInfo: \N + TaskInfo: cluster:N/A; timeout(s):14400; max_filter_ratio:0.0; priority:NORMAL + ErrorMsg: \N + CreateTimeMs: 2023-12-08 16:46:57 + FinishTimeMs: 2023-12-08 16:46:57 + TrackingUrl: +LoadStatistic: {"Unfinished backends":{},"ScannedRows":0,"TaskNumber":0,"LoadBytes":0,"All backends":{},"FileNumber":0,"FileSize":0} + User: root +1 row in set (0.05 sec) +``` +#### MV Tasks 1. 查看所有物化视图的task ```sql @@ -103,4 +158,4 @@ mysql> select * from tasks("type"="mv") where JobName="inner_mtmv_75043"; ### keywords - tasks + tasks, job, insert, mv, materilized view \ No newline at end of file diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/PAUSE-JOB.md b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/PAUSE-JOB.md index cfc88c8f6d..2a2988e067 100644 --- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/PAUSE-JOB.md +++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/PAUSE-JOB.md @@ -32,18 +32,20 @@ PAUSE JOB ### Description -用户暂停一个 JOB 作业。被停止的作业可以通过 RESUME JOB 恢复。 +用户暂停一个正在 RUNNING 状态的 JOB ，正在运行的 TASK 会被中断，JOB 状态变更为 PAUSED。 被停止的 JOB 可以通过 RESUME 操作恢复运行。 + +使用此命令需要 ADMIN 权限。 ```sql -PAUSE JOB FOR job_name; +PAUSE JOB WHERE jobname= 'jobname'; ``` ### Example -1. 暂停名称为 test1 的作业。 +1. 暂停名称为 example 的 JOB。 ```sql - PAUSE JOB FOR test1; + PAUSE JOB where jobname='example'; ``` ### Keywords diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/RESUME-JOB.md b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/RESUME-JOB.md index 2b35f8685e..79dc83562f 100644 --- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/RESUME-JOB.md +++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/RESUME-JOB.md @@ -32,23 +32,23 @@ RESUME JOB ### Description -用于重启一个 PAUSE 状态的 JOB 作业。重启的作业，将继续按照周期执行。STOP 状态的 JOB 无法被恢复。 +将处于 PAUSED 状态的 JOB 恢复为 RUNNING 状态。RUNNING 状态的 JOB 将会根据既定的调度周期去执行。 ```sql -RESUME JOB FOR job_name; +RESUME JOB where jobName = jobName; ``` ### Example -1. 重启名称为 test1 的 JOB。 +1. 恢复运行名称为 example 的 JOB。 ```sql - RESUME JOB FOR test1; + RESUME JOB where jobName= 'example'; ``` ### Keywords - RESUME, JOB + RESUME, JOB ### Best Practice diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-JOB.md b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-JOB.md index 70a90f6638..b71b4eb34d 100644 --- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-JOB.md +++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-JOB.md @@ -50,14 +50,18 @@ STOP 状态的 JOB 由用户主动触发，此时会 Cancel 正在运行中的 Finished 状态的 JOB 会保留在系统中 24 H，24H 后会被删除。 JOB 只描述作业信息， 执行会生成 TASK， TASK 状态分为 PENDING，RUNNING，SUCCEESS,FAILED,CANCELED -PENDING 表示到达触发时间了但是等待资源 RUN， 分配到资源后状态变更为RUNNING ，执行成功/失败即变更为 SUCCESS/FAILED. +PENDING 表示到达触发时间了但是等待资源 RUN， 分配到资源后状态变更为RUNNING ，执行成功/失败即变更为 SUCCESS/FAILED. CANCELED 即取消状态 ，TASK持久化最终状态，即SUCCESS/FAILED,其他状态运行中可以查到，但是如果重启则不可见。TASK只保留最新的 100 条记录。 +#### 权限 + - 目前仅支持 ***ADMIN*** 权限执行此操作。 -- 目前仅支持 ***INSERT 内表*** +#### 相关文档 [PAUSE-JOB](../Alter/PAUSE-JOB.md),[RESUME-JOB](../Alter/RESUME-JOB.md),[DROP-JOB](../Drop/DROP-JOB.md), [TVF-JOB](../../../sql-functions/table-functions/job.md), - 语法： +[TVF-TASKS](../../../sql-functions/table-functions/tasks) + +### 语法： ```sql CREATE @@ -80,7 +84,7 @@ interval: 一条有效的 Job 语句必须包含以下内容 -- 关键字 CREATE JOB 加上作业名称，它在一个 db 中标识唯一事件。 +- 关键字 CREATE JOB 加上作业名称，它在一个 db 中标识唯一事件。 JOB 名称必须是全局唯一的，如果已经存在同名的 JOB，则会报错。我们保留了 ***inner_*** 前缀在系统内部使用，因此用户不能创建以 ***inner_*** 开头的名称。 - ON SCHEDULE 子句，它指定了 Job 作业的类型和触发时间以及频率。 - DO 子句，它指定了 Job 作业触发时需要执行的操作, 即一条 SQL 语句。 @@ -95,26 +99,27 @@ CREATE JOB my_job ON SCHEDULE EVERY 1 MINUTE DO INSERT INTO db1.tbl1 SELECT * FR SCHEDULE 语句用于定义作业的执行时间，频率以及持续时间，它可以指定一次性作业或者周期性作业。 - AT timestamp - 用于一次性事件，它指定事件仅在 给定的日期和时间执行一次 timestamp，该日期和时间必须包含日期和时间 +格式：'YYYY-MM-DD HH:MM:SS', 用于一次性事件，它指定事件仅在 给定的日期和时间执行一次 timestamp，当执行完成后，JOB 状态会变更为 FINISHED。 - EVERY - 表示定期重复操作，它指定了作业的执行频率，关键字后面要指定一个时间间隔，该时间间隔可以是天、小时、分钟、秒、周。 - + 表示定期重复操作，它指定了作业的执行频率，关键字后面要指定一个时间间隔，该时间间隔可以是天、小时、分钟、秒、周。 + - interval - - 用于指定作业执行频率，它可以是天、小时、分钟、周。例如：` 1 DAY` 表示每天执行一次，` 1 HOUR` 表示每小时执行一次，` 1 MINUTE` 表示每分钟执行一次，` 1 WEEK` 表示每周执行一次。 - - STARTS timestamp + 用于指定作业执行频率，它可以是天、小时、分钟、周。例如：` 1 DAY` 表示每天执行一次，` 1 HOUR` 表示每小时执行一次，` 1 MINUTE` 表示每分钟执行一次，` 1 WEEK` 表示每周执行一次。 - 用于指定作业的开始时间，如果没有指定，则从当前时间的下一个时间点开始执行。 + - STARTS timestamp （可选字段） - - ENDS timestamp + 格式：'YYYY-MM-DD HH:MM:SS',用于指定作业的开始时间，如果没有指定，则从当前时间的下一个时间点开始执行。开始时间必须大于当前时间。 + + - ENDS timestamp （可选字段） + + 格式：'YYYY-MM-DD HH:MM:SS', 用于指定作业的结束时间，如果没有指定，则表示永久执行。该日期必须大于当前时间，如果指定了开始时间， 即 `STARTS`，则结束时间必须大于开始时间。 - 用于指定作业的结束时间，如果没有指定，则表示永久执行。 - DO - - 用于指定作业触发时需要执行的操作，目前支持所有的 ***INSERT,UPDATE*** 操作。后续我们会支持更多的操作。 + + 用于指定作业触发时需要执行的操作，目前仅支持 ***INSERT 内表*** 操作。后续我们会支持更多的操作。 ### Example @@ -137,8 +142,11 @@ CREATE JOB my_job ON SCHEDULE EVERY 1 DAY STARTS '2020-01-01 00:00:00' DO INSERT ```sql CREATE JOB my_job ON SCHEDULE EVERY 1 DAY STARTS '2020-01-01 00:00:00' ENDS '2020-01-01 00:10:00' DO INSERT INTO db1.tbl1 SELECT * FROM db2.tbl2 create_time >= days_add(now(),-1); ``` + ### INSERT JOB -目前仅支持 ***INSERT 内表*** + +- 目前仅支持 ***INSERT 内表*** +- 当下一个计划任务时间到期，即需要调度任务执行时，如果当前 JOB 仍有历史任务正在执行，则会跳过当前任务调度。因此控制一个合理的执行间隔非常重要。 ### CONFIG @@ -154,11 +162,9 @@ fe.conf ### Best Practice -合理的进行 Job 的管理，避免大量的 Job 同时触发，导致任务堆积，从而影响系统的正常运行。 -任务的执行间隔应该设置在一个合理的范围，至少应该大于任务执行时间。 +- 合理的进行 Job 的管理，避免大量的 Job 同时触发，导致任务堆积，从而影响系统的正常运行。 +- 任务的执行间隔应该设置在一个合理的范围，至少应该大于任务执行时间。 ### Keywords - CREATE, JOB - -### Best Practice \ No newline at end of file + CREATE, JOB diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/STOP-JOB.md b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Drop/DROP-JOB.md similarity index 80% rename from docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/STOP-JOB.md rename to docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Drop/DROP-JOB.md index 8a6a05b86f..a774572181 100644 --- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Alter/STOP-JOB.md +++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Drop/DROP-JOB.md @@ -1,6 +1,6 @@ --- { -"title": "STOP-JOB", +"title": "DROP-JOB", "language": "zh-CN" } --- @@ -24,26 +24,26 @@ specific language governing permissions and limitations under the License. --> -## STOP-JOB +## DROP-JOB ### Name -STOP JOB +DROP JOB ### Description -用户停止一个 JOB 作业。被停止的作业无法再重新运行。 +用户删除一个 JOB 作业。作业会被立即停止同时删除。 ```sql -STOP JOB FOR job_name; +DROP JOB where jobName='jobname'; ``` ### Example -1. 停止名称为 test1 的作业。 +1. 停止名称为 example 的作业。 ```sql - STOP JOB FOR test1; + DROP JOB where jobName='example'; ``` ### Keywords diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB-TASK.md b/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB-TASK.md deleted file mode 100644 index d1b458ea29..0000000000 --- a/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB-TASK.md +++ /dev/null @@ -1,75 +0,0 @@ ---- -{ -"title": "SHOW-JOB-TASK", -"language": "zh-CN" -} ---- - - - -## SHOW-JOB-TASK - -### Name - -SHOW JOB TASK - -### Description - -该语句用于展示 JOB 子任务的执行结果列表, 默认会保留最新的 20 条记录。 - -语法： - -```sql -SHOW JOB TASKS FOR job_name; -``` - - - -结果说明： - -``` - JobId: JobId - TaskId: TaskId - StartTime: 开始执行时间 - EndTime: 结束时间 - Status: 状态 - Result: 执行结果 - ErrMsg: 错误信息 -``` - -* State - - 有以下 2 种 State： - * SUCCESS - * FAIL - -### Example - -1. 展示名称为 test1 的 JOB 的任务执行列表 - - ```sql - SHOW JOB TASKS FOR test1; - ``` - -### Keywords - - SHOW, JOB, TASK - -### Best Practice diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md b/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md deleted file mode 100644 index 1b5da59908..0000000000 --- a/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-JOB.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -{ -"title": "SHOW-JOB", -"language": "zh-CN" -} ---- - - - -## SHOW-JOB - -### Name - -SHOW JOB - -### Description - -该语句用于展示 JOB 作业运行状态 - -语法： - -```sql -SHOW JOBS|JOB FOR job_name; -``` - -SHOW JOBS 用于展示当前 DB 下所有作业的运行状态，SHOW JOB FOR job_name 用于展示指定作业的运行状态。 - -结果说明： - -``` - Id: JobId - Db: 数据库名称 - Name: Job名称 - Definer: 创建用户 - TimeZone: 时区 - ExecuteType: RECURRING 表示循环调度，即使用 every 语句指定的调度时间，ONCE_TIME 表示一次性任务。 - ExecuteAT: ONCE_TIME 任务的执行开始时间 - ExecuteInterval: 周期调度任务的间隔 - ExecuteInterval: 周期调度任务的时间间隔单位 - STARTS: 周期调度任务设置的开始时间 - ENDS: 周期调度任务设置的结束时间 - Status: Job 状态 - LastExecuteFinishTime: 上一次执行完成时间 - ErrorMsg: 错误信息 - Comment: 备注 - - -``` - -* State - - 有以下5种State： - * RUNNING：运行中 - * PAUSED：暂停 - * STOPPED：结束（用户手动触发） - * FINISHED: 完成 - -### Example - -1. 展示当前 DB 下的所有 JOB。 - - ```sql - SHOW JOBS; - ``` - -2. 展示名称为 test1 的 JOB - - ```sql - SHOW JOB FOR test1; - ``` - -### Keywords - - SHOW, JOB - -### Best Practice