database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[fix](hdfs)(catalog) fix BE crash when hdfs-site.xml not exist in be/conf and fix compute node logic (#17244)

Browse Source 
We set LIBHDFS3_CONF env in start_be.sh, so libhdfs3 will try to read this hdfs-site.xml,
if file does not exist, it will throw error. But Doris does not handle this error, cause BE crash.
This CL mainly changes:

Modify start_be.sh to only set LIBHDFS3_CONF if hdfs-site.xml exist.
Refactor the HDFSCommonBuilder so that it can return error correctly.
Add BE IP info in status, so that we can get ip from error msg like:
ERROR 1105 (HY000): errCode = 2, detailMessage = [INTERNAL_ERROR]failed to init reader for file  000.snappy.orc, err: 
[INTERNAL_ERROR][172.21.0.101]failed to init HDFSCommonBuilder, please check check be/conf/hdfs-site.xml
The logic of prefer compute node is wrong, which causing the external table query can only assign up to 3 backends.
This CL refactor this logic and also change some FE config:

prefer_compute_node_for_external_table

If set to true, query on external table will prefer to assign to compute node.
And the max number of compute node is controlled by min_backend_num_for_external_table.
If set to false, query on external table will assign to any node.

min_backend_num_for_external_table

Only take effect when prefer_compute_node_for_external_table is true.
If the compute node number is less than this value, query on external table will try to get some mix node
to assign, to let the total number of node reach this value.
If the compute node number is larger than this value, query on external table will assign to compute node only.
This commit is contained in:
Mingyu Chen
2023-03-02 11:09:55 +08:00
committed by GitHub
parent 4682b4564c
commit 30df268c1f
24 changed files with 205 additions and 92 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
fe/fe-common/src/main/java/org/apache/doris/common/Config.java
View File

@ -1828,12 +1828,20 @@ public class Config extends ConfigBase {
public static boolean keep_scheduler_mtmv_task_when_job_deleted = false;
/**
* The candidate of the backend node for federation query such as hive table and es table query.
* If the backend of computation role is less than this value, it will acquire some mix backend.
* If the computation backend is enough, federation query will only assign to computation backend.
* If set to true, query on external table will prefer to assign to compute node.
* And the max number of compute node is controlled by min_backend_num_for_external_table.
* If set to false, query on external table will assign to any node.
*/
@ConfField(mutable = true, masterOnly = false)
public static int backend_num_for_federation = 3;
public static boolean prefer_compute_node_for_external_table = false;
/**
* Only take effect when prefer_compute_node_for_external_table is true.
* If the compute node number is less than this value, query on external table will try to get some mix node
* to assign, to let the total number of node reach this value.
* If the compute node number is larger than this value, query on external table will assign to compute node only.
*/
@ConfField(mutable = true, masterOnly = false)
public static int min_backend_num_for_external_table = 3;
/**
* Max query profile num.