database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[Improvement](multi catalog)Hive splitter. Get HDFS/S3 splits by using FileSystem api (#17706)

Browse Source 
Use FileSystem API to get splits for file in HDFS/S3 instead of calling InputFormat.getSplits.
The splits is based on blocks in HDFS/S3.
This commit is contained in:
Jibing-Li
2023-03-15 00:25:00 +08:00
committed by GitHub
parent b28f31f98d
commit 02220560c5
6 changed files with 144 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
fe/fe-common/src/main/java/org/apache/doris/common/Config.java
View File

@ -1714,6 +1714,9 @@ public class Config extends ConfigBase {
@ConfField(mutable = true, masterOnly = false)
public static long file_scan_node_split_num = 128;
@ConfField(mutable = true, masterOnly = false)
public static long file_split_size = 0; // 0 means use the block size in HDFS/S3 as split size
/**
* If set to TRUE, FE will:
* 1. divide BE into high load and low load(no mid load) to force triggering tablet scheduling;
@ -2057,6 +2060,7 @@ public class Config extends ConfigBase {
@ConfField(mutable = false, masterOnly = false)
public static String mysql_load_server_secure_path = "";
@ConfField(mutable = false, masterOnly = false)
public static int mysql_load_thread_pool = 4;
}