database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
21,892 Commits 1 Branch 0 Tags
91abfe38cdc1730f61330076cfcc616f042bee21
Commit Graph

70 Commits

Author SHA1 Message Date
wangbo
4472648c07 [branch-2.1] pick workload group usage metrics (#46177) 
pick #45284  #44870
 2024-12-31 10:09:48 +08:00
Pxl
7856662ecf [Bug](pipeline) make sink operator process eos signals after wake_up_early #45207 (#45400) 
make sink operator process eos signals after wake_up_early #45207  (#45400)
 2024-12-17 17:21:42 +08:00
Jerry Hu
efd9866b33 [chore](profile) add task state in profile (#42895) (#43013) 
## Proposed changes
pick #42895
 2024-11-01 09:51:55 +08:00
wangbo
6ceb574aa0 [branch-2.1]Pick IO limit/workload group usage table (#39839) 2024-08-23 18:51:47 +08:00
Pxl
9bb37e3616 [Bug](runtime-filter) try to fix DCHECK fail on _acquire_runtime_filter (#37805) 
## Proposed changes
pick from https://github.com/apache/doris/pull/35422
 2024-07-16 14:37:54 +08:00
Xinyi Zou
cf7595d423 [opt](memory) Optimize mem tracker accuracy (#32039) (#33140) 2024-04-10 11:42:19 +08:00
yiguolei
5cfd7c2a1c [improvement](memtracker) should counter memory usage to query when exchange sink buffer rpc (#30964) 
* [improvement](memtracker) should counter memory usage to query when rpc callback

* f

---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-02-16 10:12:25 +08:00
yiguolei
2c99c53812 [refactor](taskqueue) remove old task scheduler based wg (#30832) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-02-05 22:00:27 +08:00
zhangstar333
0e2d2f6169 [Bug](pipeline) fix pipeline task execute without wait second start rpc (#30659) 2024-02-03 20:24:58 +08:00
yiguolei
ce5ba61640 [refactor](close)Full refactor async writer (#30082) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-23 13:22:15 +08:00
zhangstar333
05a65b9f81 [improve](join) remove join probe dependency of wait rf publish finish #29792 2024-01-16 18:39:00 +08:00
wangbo
e35b26f4fc [feature](auditlog)Add runtime cpu time/peak memory metric (#29925) 2024-01-16 18:39:00 +08:00
wangbo
0d691c638b [Feature](profile)Support report runtime workload statistics #29591 2024-01-12 11:59:27 +08:00
yiguolei
738abac9ed [minor](context) duplicate query context in fragment ctx (#29364) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-01-01 22:08:23 +08:00
meiyi
1e5ff40e17 [refactor](group commit) remove future block (#27720) 
Co-authored-by: huanghaibin <284824253@qq.com>
 2023-12-11 08:41:51 +08:00
Mryange
d9d5468621 [feature](audit-log) add audit-log in insert into (#27641) 2023-11-29 15:01:57 +08:00
walter
b457856bd2 [chore](be) remove bthread scanner related codes (#27417) 2023-11-23 15:18:49 +08:00
Gabriel
fe5c4e3b46 [pipelineX](api) Fix core dump for pipelineX API (#27437) 2023-11-23 00:24:32 +08:00
meiyi
b19f275714 [improvement](insert) refactor group commit insert into (#25795) 2023-11-03 12:02:40 +08:00
wangbo
46d40b1952 [refactor](executor)Remove empty group logic #26005 2023-10-27 14:24:41 +08:00
Mryange
6b2eed779c [feature](AuditLog) add scanRows scanBytes in auditlog (#25435) 2023-10-25 10:00:35 +08:00
HappenLee
880d0d7e70 [Bug](pipeline) Support the auto partition in pipeline load (#25176) 2023-10-10 11:51:12 +08:00
HappenLee
5c020be4d2 [Bug](join) corner case cause the mark join + null aware left join core dump in regression test in pipeline query engine (#25087) 2023-10-08 22:50:12 +08:00
wangbo
7b2ff38401 query cpu hard limit based on doris scheduler (#24844) 2023-10-07 12:03:07 +08:00
HappenLee
9579634eac [Debug](pipeline) add log of pipeline scan bug (#24804) 2023-09-25 08:38:31 +08:00
Gabriel
3d786c58de [pipelineX](sink) Support async writer (#24793) 2023-09-22 23:04:23 +08:00
wangbo
26ca0b2780 Add some block counter (#24465) 
Add some block counter (#24465)
 2023-09-20 13:23:01 +08:00
HappenLee
6a33e4639a [schedule](pipeline) Remove wait schedule time in pipeline query engine and change current queue to std::mutex (#24525) 
This reverts commit 591aeaa98d1178e2e277278c7afeafef9bdb88d6.
 2023-09-18 23:57:56 +08:00
shuke
591aeaa98d Revert "[schedule](pipeline) Remove wait schedule time in pipeline query engine (#23994)" (#24472) 
This reverts commit 32a7eef96a09799c8336c1964bfe7d676b7e4c98.
 2023-09-18 09:57:38 +08:00
zclllyybb
d3f1388717 [Feature](partitions) Support auto-partition (#24153) 
Co-authored-by: zhangstar333 <2561612514@qq.com>
 2023-09-12 15:23:15 +08:00
HappenLee
32a7eef96a [schedule](pipeline) Remove wait schedule time in pipeline query engine (#23994) 
Co-authored-by: yiguolei <676222867@qq.com>
 2023-09-10 17:06:51 +08:00
meiyi
82dc970916 [feature](insert) Support group commit insert (#22829) 2023-09-08 15:51:03 +08:00
zclllyybb
fdb7a44f57 Revert "[Feature](partitions) Support auto partition" (#24024) 
* Revert "[Feature](partitions) Support auto partition (#23236)"

This reverts commit 6c544dd2011d731b8c9c51384c77bcf19c017981.

* Update config.h
 2023-09-07 17:08:26 +08:00
zclllyybb
6c544dd201 [Feature](partitions) Support auto partition (#23236) 
Co-authored-by: zhangstar333 <2561612514@qq.com>
 2023-09-06 16:26:45 +08:00
HappenLee
6d41272421 [Opt](pipeline) Refactor the short circuit of join pipeline (#23639) 
* [Opt](pipeline) Refactor the short circuit of join pipeline

* change core by cr
 2023-08-30 18:44:14 +08:00
Gabriel
9d1f2cd8e0 [Improvement](pipeline) Terminate early for short-circuit join (#23378) 2023-08-23 19:40:17 +08:00
yiguolei
bcdb481374 [refactor](fragment) refactor non pipeline fragment executor (#23281) 
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2023-08-22 16:00:34 +08:00
Gabriel
dcd6c3c022 [pipelineX](refactor) propose a new pipeline execution model (#22562) 2023-08-21 15:38:45 +08:00
HappenLee
3d4ec1ac88 [pipeline](exec) support async writer in jdbc sink in pipeline query engine (#23144) 
support async writer in jdbc sink in pipeline query engine
 2023-08-18 17:07:57 +08:00
Gabriel
23e7423748 [pipeline](refactor) refactor pipeline task schedule logics (#22028) 2023-07-25 17:18:26 +08:00
wangbo
37ca133845 [feature](profile)add monotonice timer for pipeline task #21791 
Add monotonice timer for piplinetask;

WaitBfTime

Task1BeginExecuteTime
Task2EosTime
Task3SrcPendingFinishOverTime
Task4DstPendingFinishOverTime
Task5TotalTime
Task6ClosePipelineTime
 2023-07-18 07:57:14 +08:00
zhangstar333
29146c680e [refactor](profile)add node id info in pipeline profile (#21823) 2023-07-17 15:24:02 +08:00
HappenLee
307149dc35 [pipeline](task_queue) remove disable steal in task queue to speed up query (#21692) 
TPCH Q9

before: 2.74s
after: 2.33s
 2023-07-10 22:21:56 +08:00
Xinyi Zou
b86dd11a7d [fix](pipeline) refactor olap table sink close (#20771) 
For pipeline, olap table sink close is divided into three stages, try_close() --> pending_finish() --> close()
only after all node channels are done or canceled, pending_finish() will return false, close() will start.
this will avoid block pipeline on close().

In close, check the index channel intolerable failure status after each node channel failure,
if intolerable failure is true, the close will be terminated in advance, and all node channels will be canceled to avoid meaningless blocking.
 2023-07-04 11:27:51 +08:00
Lijia Liu
76bdcf1d26 [improvement](pipeline) task group scan entity (#19924) 2023-06-25 14:43:35 +08:00
Pxl
db1da6b787 [Chore](pipeline) add some profile log when pipeline canceled (#20825) 
add some profile log when pipeline canceled
 2023-06-16 10:54:54 +08:00
Pxl
4bfceb7acb [Bug](pipeline) fix pipeline task call finish_p_dependency more than once (#20851) 
fix pipeline task call finish_p_dependency more than once

When pipeline task meet eos->PENDING_FINISH->CANCELED, this task will call finish_p_dependency twice.
 2023-06-15 19:21:40 +08:00
Pxl
5fd9f58bd3 [Chore](pipeline-engine) adjus queryt canceled log on pipeline engine (#20702) 
adjus queryt canceled log on pipeline engine
 2023-06-12 18:23:19 +08:00
TengJianPing
1b94b6368f [fix](load) in strict mode, return error for insert if datatype convert fails (#20378) 
* [fix](load) in strict mode, return error for load and insert if datatype convert fails

Revert "[fix](MySQL) the way Doris handles boolean type is consistent with MySQL (#19416)"

This reverts commit 68eb420cabe5b26b09d6d4a2724ae12699bdee87.

Since it changed other behaviours, e.g. in strict mode insert into t_int values ("a"),
it will result 0 is inserted into table, but it should return error instead.

* fix be ut

* fix regression tests
 2023-06-06 12:04:03 +08:00
HappenLee
e5b0d7a5cd [CTE](eof) Support cte reuse reduce counter by eof status and pipeline task mem can release (#20056) 2023-05-26 22:03:29 +08:00