[improvememt](file-cache) increase virtual node number to make file cache more even (#24143)

The origin virtual number is Math.max(Math.min(512 / backends.size(), 32), 2);, which is too small,
causing uneven cache distribution when enabling file cache.
This commit is contained in:
Mingyu Chen
2023-09-10 19:56:53 +08:00
committed by GitHub
parent 102abff071
commit 1df2e4454f
2 changed files with 9 additions and 2 deletions

View File

@ -2157,4 +2157,12 @@ public class Config extends ConfigBase {
"是否禁止LocalDeployManager删除节点",
"Whether to disable LocalDeployManager drop node"})
public static boolean disable_local_deploy_manager_drop_node = true;
@ConfField(mutable = true, description = {
"开启 file cache 后,一致性哈希算法中,每个节点的虚拟节点数。"
+ "该值越大,哈希算法的分布越均匀,但是会增加内存开销。",
"When file cache is enabled, the number of virtual nodes of each node in the consistent hash algorithm. "
+ "The larger the value, the more uniform the distribution of the hash algorithm, "
+ "but it will increase the memory overhead."})
public static int virtual_node_number = 2048;
}

View File

@ -96,9 +96,8 @@ public class FederationBackendPolicy {
throw new UserException("No available backends");
}
backendMap.putAll(backends.stream().collect(Collectors.groupingBy(Backend::getHost)));
int virtualNumber = Math.max(Math.min(512 / backends.size(), 32), 2);
consistentHash = new ConsistentHash<>(Hashing.murmur3_128(), new ScanRangeHash(),
new BackendHash(), backends, virtualNumber);
new BackendHash(), backends, Config.virtual_node_number);
}
public Backend getNextBe() {