[opt](routine load) optimize routine load timeout logic (#40818) (#41135)

pick #40818

If IO/CPU resources are tight, routine load task is likely to timeout.
The current method is self-adaption backoff
https://github.com/apache/doris/pull/32227, but the problem is it will
do some ineffective work to match proper timeout. For one routine load
task, a better way to handle task is finishing executing instead of
retry when resources are tight. Therefore, this pr increase timeout to
make "task always finish even if it is slow when resources are tight".
This commit is contained in:
hui lai
2024-09-25 14:14:02 +08:00
committed by GitHub
parent 386c55f00a
commit 11bad4cbc9
8 changed files with 21 additions and 66 deletions

View File

@ -1198,6 +1198,12 @@ public class Config extends ConfigBase {
@ConfField(mutable = true, masterOnly = true)
public static int max_routine_load_task_num_per_be = 1024;
/**
* routine load timeout is equal to maxBatchIntervalS * routine_load_task_timeout_multiplier.
*/
@ConfField(mutable = true, masterOnly = true)
public static int routine_load_task_timeout_multiplier = 10;
/**
* the max timeout of get kafka meta.
*/