database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[opt](nereids) enhance broadcast join cost calculation (#22092)

Browse Source 
Enhance broadcast join cost calculation, by considering both the build side effort from building bigger hash table, and more probe side effort from bigger cost of ProbeWhenBuildSideOutput and ProbeWhenSearchHashTable, if parallel_fragment_exec_instance_num is more than 1.

Current solution gives a penalty factor on rightRowCount, and the factor is the total instance number to the power of 2.
Penalty on outputRows is not taken currently and will be refined in next generation cost model.

Also brings some update for shape checking:

update original control variable in shape file parallel_fragment_exec_instance_num to parallel_pipeline_task_num, if pipeline is enabled.
fix a be_number variable inactive issue.
This commit is contained in:
xzj7019
2023-07-28 23:06:02 +08:00
committed by GitHub
parent ebc8988f70
commit f7c106c709
192 changed files with 880 additions and 766 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java
View File

@ -69,6 +69,9 @@ class CostModelV1 extends PlanVisitor<Cost, PlanContext> {
public CostModelV1() {
if (ConnectContext.get().getSessionVariable().isPlayNereidsDump()) {
// TODO: @bingfeng refine minidump setting, and pass testMinidumpUt
beNumber = 1;
} else if (ConnectContext.get().getSessionVariable().getBeNumber() != -1) {
beNumber = ConnectContext.get().getSessionVariable().getBeNumber();
} else {
beNumber = Math.max(1, ConnectContext.get().getEnv().getClusterInfo().getBackendsNumber(true));
@ -284,8 +287,24 @@ class CostModelV1 extends PlanVisitor<Cost, PlanContext> {
}
if (context.isBroadcastJoin()) {
double broadcastJoinPenalty = broadCastJoinBalancePenalty(probeStats, buildStats);
return CostV1.of(leftRowCount * broadcastJoinPenalty + rightRowCount + outputRowCount,
// compared with shuffle join, bc join will be taken a penalty for both build and probe side;
// currently we use the following factor as the penalty factor:
// build side factor: totalInstanceNumber to the power of 2, standing for the additional effort for
// bigger cost for building hash table, taken on rightRowCount
// probe side factor: totalInstanceNumber to the power of 2, standing for the additional effort for
// bigger cost for ProbeWhenBuildSideOutput effort and ProbeWhenSearchHashTableTime
// on the output rows, taken on outputRowCount()
double probeSideFactor = 1.0;
double buildSideFactor = ConnectContext.get().getSessionVariable().getBroadcastRightTableScaleFactor();
int parallelInstance = Math.max(1, ConnectContext.get().getSessionVariable().getParallelExecInstanceNum());
int totalInstanceNumber = parallelInstance * beNumber;
if (buildSideFactor <= 1.0) {
// use totalInstanceNumber to the power of 2 as the default factor value
buildSideFactor = Math.pow(totalInstanceNumber, 0.5);
}
// TODO: since the outputs rows may expand a lot, penalty on it will cause bc never be chosen.
// will refine this in next generation cost model.
return CostV1.of(leftRowCount + rightRowCount * buildSideFactor + outputRowCount * probeSideFactor,
rightRowCount,
0,
0

2
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
View File

@ -810,7 +810,7 @@ public class SessionVariable implements Serializable, Writable {
public boolean enableDpHypTrace = false;
@VariableMgr.VarAttr(name = BROADCAST_RIGHT_TABLE_SCALE_FACTOR)
private double broadcastRightTableScaleFactor = 10.0;
private double broadcastRightTableScaleFactor = 0.0;
@VariableMgr.VarAttr(name = BROADCAST_ROW_COUNT_LIMIT, needForward = true)
private double broadcastRowCountLimit = 30000000;