[fix](dynamic-partition) fix wrong check of replication num (#13755)

2022-11-02 12:55:33 +08:00
parent 667cfe5598
commit d5becdb4a1
8 changed files with 204 additions and 25 deletions
--- a/docs/en/docs/advanced/partition/dynamic-partition.md
+++ b/docs/en/docs/advanced/partition/dynamic-partition.md
@ -459,9 +459,90 @@ When dynamic partitioning feature is disabled, Doris will no longer manage parti

 1. After creating the dynamic partition table, it prompts ```Could not create table with dynamic partition when fe config dynamic_partition_enable is false```

-         Because the main switch of dynamic partition, that is, the configuration of FE ```dynamic_partition_enable``` is false, the dynamic partition table cannot be created.
+	Because the main switch of dynamic partition, that is, the configuration of FE ```dynamic_partition_enable``` is false, the dynamic partition table cannot be created.
         
-         At this time, please modify the FE configuration file, add a line ```dynamic_partition_enable=true```, and restart FE. Or execute the command ADMIN SET FRONTEND CONFIG ("dynamic_partition_enable" = "true") to turn on the dynamic partition switch.
+	At this time, please modify the FE configuration file, add a line ```dynamic_partition_enable=true```, and restart FE. Or execute the command ADMIN SET FRONTEND CONFIG ("dynamic_partition_enable" = "true") to turn on the dynamic partition switch.
+
+2. Replica settings for dynamic partitions
+
+    Dynamic partitions are automatically created by scheduling logic inside the system. When creating a partition automatically, the partition properties (including the number of replicas of the partition, etc.) are all prefixed with `dynamic_partition`, rather than the default properties of the table. for example:
+
+    ```
+    CREATE TABLE tbl1 (
+    `k1` int,
+    `k2` date
+    )
+    PARTITION BY RANGE(k2)()
+    DISTRIBUTED BY HASH(k1) BUCKETS 3
+    PROPERTIES
+    (
+    "dynamic_partition.enable" = "true",
+    "dynamic_partition.time_unit" = "DAY",
+    "dynamic_partition.end" = "3",
+    "dynamic_partition.prefix" = "p",
+    "dynamic_partition.buckets" = "32",
+    "dynamic_partition.replication_num" = "1",
+    "dynamic_partition.start" = "-3",
+    "replication_num" = "3"
+    );
+    ```
+
+    In this example, no initial partition is created (partition definition in PARTITION BY clause is empty), and `DISTRIBUTED BY HASH(k1) BUCKETS 3`, `"replication_num" = "3"`, `"dynamic_partition is set. replication_num" = "1` and `"dynamic_partition.buckets" = "32"`.
+
+    We make the first two parameters the default parameters for the table, and the last two parameters the dynamic partition-specific parameters.
+
+    When the system automatically creates a partition, it will use the two configurations of bucket number 32 and replica number 1 (that is, parameters dedicated to dynamic partitions). Instead of the two configurations of bucket number 3 and replica number 3.
+
+    When a user manually adds a partition through the `ALTER TABLE tbl1 ADD PARTITION` statement, the two configurations of bucket number 3 and replica number 3 (that is, the default parameters of the table) will be used.
+
+    That is, dynamic partitioning uses a separate set of parameter settings. The table's default parameters are used only if no dynamic partition-specific parameters are set. as follows:
+
+    ```
+    CREATE TABLE tbl2 (
+    `k1` int,
+    `k2` date
+    )
+    PARTITION BY RANGE(k2)()
+    DISTRIBUTED BY HASH(k1) BUCKETS 3
+    PROPERTIES
+    (
+    "dynamic_partition.enable" = "true",
+    "dynamic_partition.time_unit" = "DAY",
+    "dynamic_partition.end" = "3",
+    "dynamic_partition.prefix" = "p",
+    "dynamic_partition.start" = "-3",
+    "dynamic_partition.buckets" = "32",
+    "replication_num" = "3"
+    );
+    ```
+
+    In this example, if `dynamic_partition.replication_num` is not specified separately, the default parameter of the table is used, which is `"replication_num" = "3"`.
+
+	And the following example:
+
+     ```
+     CREATE TABLE tbl3 (
+     `k1` int,
+     `k2` date
+     )
+     PARTITION BY RANGE(k2)(
+         PARTITION p1 VALUES LESS THAN ("2019-10-10")
+     )
+     DISTRIBUTED BY HASH(k1) BUCKETS 3
+     PROPERTIES
+     (
+     "dynamic_partition.enable" = "true",
+     "dynamic_partition.time_unit" = "DAY",
+     "dynamic_partition.end" = "3",
+     "dynamic_partition.prefix" = "p",
+     "dynamic_partition.start" = "-3",
+     "dynamic_partition.buckets" = "32",
+     "dynamic_partition.replication_num" = "1",
+     "replication_num" = "3"
+     );
+     ```
+
+     In this example, there is a manually created partition p1. This partition will use the default settings for the table, which are 3 buckets and 3 replicas. The dynamic partitions automatically created by the subsequent system will still use the special parameters for dynamic partitions, that is, the number of buckets is 32 and the number of replicas is 1.

 ## More Help

--- a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
+++ b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
@ -577,7 +577,7 @@ distribution_desc
        "dynamic_partition.end" = "3",
        "dynamic_partition.prefix" = "p",
        "dynamic_partition.buckets" = "32",
-        "dynamic_partition."replication_allocation" = "tag.location.group_a:3"
+        "dynamic_partition.replication_allocation" = "tag.location.group_a:3"
     );
    ```

--- a/docs/zh-CN/docs/advanced/partition/dynamic-partition.md
+++ b/docs/zh-CN/docs/advanced/partition/dynamic-partition.md
@ -453,6 +453,87 @@ mysql> SHOW DYNAMIC PARTITION TABLES;

   这时候请修改 FE 的配置文件，增加一行 `dynamic_partition_enable=true`，并重启 FE。或者执行命令 ADMIN SET FRONTEND CONFIG ("dynamic_partition_enable" = "true") 将动态分区开关打开即可。

+2. 关于动态分区的副本设置
+
+    动态分区是由系统内部的调度逻辑自动创建的。在自动创建分区时，所使用的分区属性（包括分区的副本数等），都是单独使用 `dynamic_partition` 前缀的属性，而不是使用表的默认属性。举例说明：
+
+    ```
+    CREATE TABLE tbl1 (
+    `k1` int,
+    `k2` date
+    )
+    PARTITION BY RANGE(k2)()
+    DISTRIBUTED BY HASH(k1) BUCKETS 3
+    PROPERTIES
+    (
+    "dynamic_partition.enable" = "true",
+    "dynamic_partition.time_unit" = "DAY",
+    "dynamic_partition.end" = "3",
+    "dynamic_partition.prefix" = "p",
+    "dynamic_partition.buckets" = "32",
+    "dynamic_partition.replication_num" = "1",
+    "dynamic_partition.start" = "-3",
+    "replication_num" = "3"
+    );
+    ```
+
+    这个示例中，没有创建任何初始分区（PARTITION BY 子句中的分区定义为空），并且设置了 `DISTRIBUTED BY HASH(k1) BUCKETS 3`, `"replication_num" = "3"`, `"dynamic_partition.replication_num" = "1"` 和 `"dynamic_partition.buckets" = "32"`。
+
+    我们将前两个参数成为表的默认参数，而后两个参数成为动态分区专用参数。
+
+    当系统自动创爱分区时，会使用分桶数 32 和 副本数 1 这两个配置（即动态分区专用参数）。而不是分桶数 3 和 副本数 3 这两个配置。
+
+    当用户通过 `ALTER TABLE tbl1 ADD PARTITION` 语句手动添加分区时，则会使用分桶数 3 和 副本数 3 这两个配置（即表的默认参数）。
+
+    即动态分区使用一套独立的参数设置。只有当没有设置动态分区专用参数时，才会使用表的默认参数。如下：
+
+    ```
+    CREATE TABLE tbl2 (
+    `k1` int,
+    `k2` date
+    )
+    PARTITION BY RANGE(k2)()
+    DISTRIBUTED BY HASH(k1) BUCKETS 3
+    PROPERTIES
+    (
+    "dynamic_partition.enable" = "true",
+    "dynamic_partition.time_unit" = "DAY",
+    "dynamic_partition.end" = "3",
+    "dynamic_partition.prefix" = "p",
+    "dynamic_partition.start" = "-3",
+    "dynamic_partition.buckets" = "32",
+    "replication_num" = "3"
+    );
+    ```
+    
+    这个示例中，没有单独指定 `dynamic_partition.replication_num`，则动态分区会使用表的默认参数，即 `"replication_num" = "3"`。
+
+    而如下示例：
+
+    ```
+    CREATE TABLE tbl3 (
+    `k1` int,
+    `k2` date
+    )
+    PARTITION BY RANGE(k2)(
+        PARTITION p1 VALUES LESS THAN ("2019-10-10")
+    )
+    DISTRIBUTED BY HASH(k1) BUCKETS 3
+    PROPERTIES
+    (
+    "dynamic_partition.enable" = "true",
+    "dynamic_partition.time_unit" = "DAY",
+    "dynamic_partition.end" = "3",
+    "dynamic_partition.prefix" = "p",
+    "dynamic_partition.start" = "-3",
+    "dynamic_partition.buckets" = "32",
+    "dynamic_partition.replication_num" = "1",
+    "replication_num" = "3"
+    );
+    ```
+
+    这个示例中，有一个手动创建的分区 p1。这个分区会使用表的默认设置，即分桶数 3 和副本数 3。而后续系统自动创建的动态分区，依然会使用动态分区专用参数，即分桶数 32 和副本数 1。
+
 ## 更多帮助

 关于动态分区使用的更多详细语法及最佳实践，请参阅 [SHOW DYNAMIC PARTITION](../../sql-manual/sql-reference/Show-Statements/SHOW-DYNAMIC-PARTITION.md) 命令手册，你也可以在 MySql 客户端命令行下输入 `HELP ALTER TABLE` 获取更多帮助信息。
--- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
+++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md
@ -578,7 +578,7 @@ distribution_desc
        "dynamic_partition.end" = "3",
        "dynamic_partition.prefix" = "p",
        "dynamic_partition.buckets" = "32",
-        "dynamic_partition."replication_allocation" = "tag.location.group_a:3"
+        "dynamic_partition.replication_allocation" = "tag.location.group_a:3"
     );
    ```

--- a/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/DynamicPartitionUtil.java
@ -211,7 +211,7 @@ public class DynamicPartitionUtil {
        }
    }

-    private static void checkReplicationNum(String val) throws DdlException {
+    private static void checkReplicationNum(String val, Database db) throws DdlException {
        if (Strings.isNullOrEmpty(val)) {
            throw new DdlException("Invalid properties: " + DynamicPartitionProperty.REPLICATION_NUM);
        }
@ -222,6 +222,8 @@ public class DynamicPartitionUtil {
        } catch (NumberFormatException e) {
            ErrorReport.reportDdlException(ErrorCode.ERROR_DYNAMIC_PARTITION_REPLICATION_NUM_FORMAT, val);
        }
+        ReplicaAllocation replicaAlloc = new ReplicaAllocation(Short.valueOf(val));
+        Env.getCurrentSystemInfo().selectBackendIdsForReplicaCreation(replicaAlloc, db.getClusterName(), null);
    }

    private static void checkReplicaAllocation(ReplicaAllocation replicaAlloc, Database db) throws DdlException {
@ -585,19 +587,18 @@ public class DynamicPartitionUtil {
            analyzedProperties.put(DynamicPartitionProperty.TIME_ZONE, val);
        }

-        if (properties.containsKey(DynamicPartitionProperty.REPLICATION_NUM)) {
-            String val = properties.get(DynamicPartitionProperty.REPLICATION_NUM);
-            checkReplicationNum(val);
-            properties.remove(DynamicPartitionProperty.REPLICATION_NUM);
-            analyzedProperties.put(DynamicPartitionProperty.REPLICATION_ALLOCATION,
-                    new ReplicaAllocation(Short.valueOf(val)).toCreateStmt());
-        }
-
+        // check replication_allocation first, then replciation_num
        if (properties.containsKey(DynamicPartitionProperty.REPLICATION_ALLOCATION)) {
            ReplicaAllocation replicaAlloc = PropertyAnalyzer.analyzeReplicaAllocation(properties, "dynamic_partition");
            checkReplicaAllocation(replicaAlloc, db);
            properties.remove(DynamicPartitionProperty.REPLICATION_ALLOCATION);
            analyzedProperties.put(DynamicPartitionProperty.REPLICATION_ALLOCATION, replicaAlloc.toCreateStmt());
+        } else if (properties.containsKey(DynamicPartitionProperty.REPLICATION_NUM)) {
+            String val = properties.get(DynamicPartitionProperty.REPLICATION_NUM);
+            checkReplicationNum(val, db);
+            properties.remove(DynamicPartitionProperty.REPLICATION_NUM);
+            analyzedProperties.put(DynamicPartitionProperty.REPLICATION_ALLOCATION,
+                    new ReplicaAllocation(Short.valueOf(val)).toCreateStmt());
        } else {
            checkReplicaAllocation(olapTable.getDefaultReplicaAllocation(), db);
        }
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java
@ -108,6 +108,28 @@ public class DynamicPartitionTableTest {
                Env.getCurrentInternalCatalog().getDbOrAnalysisException("default_cluster:test");
        OlapTable table = (OlapTable) db.getTableOrAnalysisException("dynamic_partition_normal");
        Assert.assertTrue(table.getTableProperty().getDynamicPartitionProperty().getReplicaAllocation().isNotSet());
+
+        // test only set dynamic_partition.replication_num
+        createOlapTblStmt = "CREATE TABLE test.`dynamic_partition_normal2` (\n"
+                + "`uuid` varchar(255) NULL,\n"
+                + "`action_datetime` date NULL\n"
+                + ")\n"
+                + "DUPLICATE KEY(uuid)\n"
+                + "PARTITION BY RANGE(action_datetime)()\n"
+                + "DISTRIBUTED BY HASH(uuid) BUCKETS 3\n"
+                + "PROPERTIES\n"
+                + "(\n"
+                + "\"dynamic_partition.enable\" = \"true\",\n"
+                + "\"dynamic_partition.time_unit\" = \"DAY\",\n"
+                + "\"dynamic_partition.end\" = \"3\",\n"
+                + "\"dynamic_partition.prefix\" = \"p\",\n"
+                + "\"dynamic_partition.buckets\" = \"32\",\n"
+                + "\"dynamic_partition.replication_num\" = \"1\",\n"
+                + "\"dynamic_partition.create_history_partition\"=\"true\",\n"
+                + "\"dynamic_partition.start\" = \"-3\"\n"
+                + ");\n"
+                + "\n";
+        createTable(createOlapTblStmt);
    }

    @Test
@ -431,23 +453,17 @@ public class DynamicPartitionTableTest {
                + "PROPERTIES (\n" + "\"replication_num\" = \"1\",\n" + "\"dynamic_partition.enable\" = \"true\",\n"
                + "\"dynamic_partition.start\" = \"-3\",\n" + "\"dynamic_partition.end\" = \"3\",\n"
                + "\"dynamic_partition.time_unit\" = \"day\",\n" + "\"dynamic_partition.prefix\" = \"p\",\n"
-                + "\"dynamic_partition.buckets\" = \"1\",\n" + "\"dynamic_partition.replication_num\" = \"2\"\n" + ");";
+                + "\"dynamic_partition.buckets\" = \"1\",\n" + "\"dynamic_partition.replication_num\" = \"1\"\n" + ");";
        createTable(createOlapTblStmt);
        Database db =
                Env.getCurrentInternalCatalog().getDbOrAnalysisException("default_cluster:test");
        OlapTable table = (OlapTable) db.getTableOrAnalysisException(tableName);
-        Assert.assertEquals(2,
-                table.getTableProperty().getDynamicPartitionProperty().getReplicaAllocation().getTotalReplicaNum());
-
-        String alter1 =
-                "alter table test.dynamic_partition_replication_num set ('dynamic_partition.replication_num' = '1')";
-        ExceptionChecker.expectThrowsNoException(() -> alterTable(alter1));
        Assert.assertEquals(1,
                table.getTableProperty().getDynamicPartitionProperty().getReplicaAllocation().getTotalReplicaNum());

-        String alter2 =
+        String alter1 =
                "alter table test.dynamic_partition_replication_num set ('dynamic_partition.replication_num' = '0')";
-        ExceptionChecker.expectThrows(AnalysisException.class, () -> alterTable(alter2));
+        ExceptionChecker.expectThrows(AnalysisException.class, () -> alterTable(alter1));
        Assert.assertEquals(1,
                table.getTableProperty().getDynamicPartitionProperty().getReplicaAllocation().getTotalReplicaNum());
    }
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/ModifyBackendTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/ModifyBackendTest.java
@ -96,7 +96,7 @@ public class ModifyBackendTest {
                + "buckets 3 properties(\n" + "    \"dynamic_partition.enable\" = \"true\",\n"
                + "    \"dynamic_partition.time_unit\" = \"DAY\",\n" + "    \"dynamic_partition.start\" = \"-3\",\n"
                + "    \"dynamic_partition.end\" = \"3\",\n" + "    \"dynamic_partition.prefix\" = \"p\",\n"
-                + "    \"dynamic_partition.buckets\" = \"1\",\n" + "    \"dynamic_partition.replication_num\" = \"1\"\n"
+                + "    \"dynamic_partition.buckets\" = \"1\",\n" + "    \"dynamic_partition.replication_num\" = \"3\"\n"
                + ");";
        CreateTableStmt createStmt3 = (CreateTableStmt) UtFrameUtils.parseAndAnalyzeStmt(createStr, connectContext);
        //partition create failed, because there is no BE with "default" tag
--- a/fe/fe-core/src/test/java/org/apache/doris/utframe/DorisAssert.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/utframe/DorisAssert.java
@ -127,7 +127,7 @@ public class DorisAssert {
        Env.getCurrentEnv().createMaterializedView(createMaterializedViewStmt);
        checkAlterJob();
        // waiting table state to normal
-        Thread.sleep(100);
+        Thread.sleep(1000);
        return this;
    }

@ -137,7 +137,7 @@ public class DorisAssert {
        Env.getCurrentEnv().alterTable(alterTableStmt);
        checkAlterJob();
        // waiting table state to normal
-        Thread.sleep(100);
+        Thread.sleep(1000);
        return this;
    }