database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[branch-2.1](routine-load) increase routine load job default max batch size and rows (#37388)

Browse Source 
pick #36632

Most users only care about the size of **max_batch_interval**, but in
order to achieve an interval effect, they have to configure
**max_batch_rows** and **max_batch_size** according to the
characteristics of the data. By adjusting these two default values,
users do not need to worry about configuration in most scenarios.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
This commit is contained in:
hui lai
2024-07-07 18:35:08 +08:00
committed by GitHub
parent a05406ecc9
commit 97e4025ee0
2 changed files with 3 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
View File

@ -109,8 +109,8 @@ public abstract class RoutineLoadJob extends AbstractTxnStateChangeCallback impl
public static final double DEFAULT_MAX_FILTER_RATIO = 1.0;
public static final long DEFAULT_MAX_INTERVAL_SECOND = 10;
public static final long DEFAULT_MAX_BATCH_ROWS = 200000;
public static final long DEFAULT_MAX_BATCH_SIZE = 100 * 1024 * 1024; // 100MB
public static final long DEFAULT_MAX_BATCH_ROWS = 20000000;
public static final long DEFAULT_MAX_BATCH_SIZE = 1024 * 1024 * 1024; // 1GB
public static final long DEFAULT_EXEC_MEM_LIMIT = 2 * 1024 * 1024 * 1024L;
public static final boolean DEFAULT_STRICT_MODE = false; // default is false
public static final int DEFAULT_SEND_BATCH_PARALLELISM = 1;