Commit Graph

12 Commits

Author SHA1 Message Date
59261174d5 [chore](unused) Remove unused variable CPU_HARD_LIMIT in task_group.cc (#25076)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-10-07 03:36:13 -05:00
7b2ff38401 query cpu hard limit based on doris scheduler (#24844) 2023-10-07 12:03:07 +08:00
c9b2f4cb92 [workload](pipeline) Add cgroup cpu controller (#24052) 2023-09-21 21:49:33 +08:00
76bdcf1d26 [improvement](pipeline) task group scan entity (#19924) 2023-06-25 14:43:35 +08:00
f9dfcb923d [Enhancement] Change Create Resource Group Grammar (#20249) 2023-05-31 15:23:24 +08:00
b81e9e2521 [fix](resource-group) Fix resource group memory isolation may release too much memory (#20066)
Suppose three queries are executed in a resource group with a memory_limit of 8G, and they consume memory of query_a = 3G, query_b = 3G, and query_c = 3G. The total memory used is counted as 9G when the resource group GC is executed, which exceeds the resource group limit and cancels query_a.

When the resource group is next GC, the memory of query_a may not be freed yet, and it will be counted again in the total memory consumed by that resource group, which again exceeds the resource group limit and cancels query_b.

From the user's perspective, it is fine to execute query_a and query_b at the same time, but executing query_ a, query_b and query_c will be cancelled for two queries, which is not as expected.

This pr skips the queries that are cancelled when counting the memory used by the resource group. If this causes the process memory to grow, the process gc will handle it.
2023-05-26 14:50:12 +08:00
33fd965b5c [feature-wip](resouce-group) Supports memory soft isolation of resource group (#19802)
create resource groups name properties(
    'enable_memory_overcommit' = 'true' // whether to enable memory soft isolation
)
2023-05-21 19:33:57 +08:00
6c9c9e9765 [feature-wip](resource-group) Supports memory hard isolation of resource group (#19526) 2023-05-15 22:45:46 +08:00
38294b98db Fix comparator of ResouceGroupSet (#19523) 2023-05-13 09:17:16 +08:00
e412dd12e8 [chore](build) Use include-what-you-use to optimize includes (PART II) (#18761)
Currently, there are some useless includes in the codebase. We can use a tool named include-what-you-use to optimize these includes. By using a strict include-what-you-use policy, we can get lots of benefits from it.
2023-04-19 23:11:48 +08:00
81799d614e [feature-wip](resource-group) support resource group interface in be. (#18588) 2023-04-14 14:00:49 +08:00
2ee1468576 [improvement](executor) Support task group schedule in pipeline engine (#17615) 2023-03-30 10:49:50 +08:00