diff --git a/docs/en/docs/advanced/partition/auto-partition.md b/docs/en/docs/advanced/partition/auto-partition.md index 0d84f83630..4117a0d90a 100644 --- a/docs/en/docs/advanced/partition/auto-partition.md +++ b/docs/en/docs/advanced/partition/auto-partition.md @@ -197,3 +197,4 @@ A partition created by the AUTO PARTITION function has the exact same functional - Tables that use AUTO PARTITION only have their partitions created automatically instead of manually. The original use of the table and the partitions it creates is the same as for non-AUTO PARTITION tables or partitions. - To prevent accidental creation of too many partitions, we use the [FE Configuration](../../admin-manual/config/fe-config) `max_auto_partition_num` controls the maximum number of partitions an AUTO PARTITION table can hold. This value can be adjusted if necessary - When importing data to a table with AUTO PARTITION enabled, the polling interval for data sent by the Coordinator is different from that of a normal table. For details, see `olap_table_sink_send_interval_auto_partition_factor` in [BE Configuration](../../admin-manual/config/be-config). +- In the case of inserting data with [insert-overwrite](../../sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT-OVERWRITE) command, if an overwrite partition is specified, the AUTO PARTITION table behaves as a normal table during this process and no new partitions will be created. diff --git a/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT-OVERWRITE.md b/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT-OVERWRITE.md index 8ba6d71e41..f94e37f5e0 100644 --- a/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT-OVERWRITE.md +++ b/docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT-OVERWRITE.md @@ -68,9 +68,9 @@ INSERT OVERWRITE table table_name Notice: -In the current version, the session variable `enable_insert_strict` is set to `true` by default. If some data that does not conform to the format of the target table is filtered out during the execution of the `INSERT OVERWRITE` statement, such as when overwriting a partition and not all partition conditions are satisfied, overwriting the target table will fail. - -The `INSERT OVERWRITE` statement first creates a new table, inserts the data to be overwritten into the new table, and then atomically replaces the old table with the new table and modifies its name. Therefore, during the process of overwriting the table, the data in the old table can still be accessed normally until the overwriting is completed. +1. In the current version, the session variable `enable_insert_strict` is set to `true` by default. If some data that does not conform to the format of the target table is filtered out during the execution of the `INSERT OVERWRITE` statement, such as when overwriting a partition and not all partition conditions are satisfied, overwriting the target table will fail. +2. If the target table of the INSERT OVERWRITE is an [AUTO-PARTITION-table](../../../../advanced/partition/auto-partition), then new partitions can be created if PARTITION is not specified (that is, rewrite the whole table). If PARTITION for overwrite is specified, then the AUTO PARTITION table behaves as if it were a normal partitioned table during this process, and data that does not satisfy the existing partition conditions is filtered instead of creating a new partition. +3. The `INSERT OVERWRITE` statement first creates a new table, inserts the data to be overwritten into the new table, and then atomically replaces the old table with the new table and modifies its name. Therefore, during the process of overwriting the table, the data in the old table can still be accessed normally until the overwriting is completed. ### Example @@ -177,4 +177,4 @@ PROPERTIES ( ### Keywords - INSERT OVERWRITE + INSERT OVERWRITE, OVERWRITE diff --git a/docs/zh-CN/docs/advanced/partition/auto-partition.md b/docs/zh-CN/docs/advanced/partition/auto-partition.md index 185b1f3b9e..c998b38db4 100644 --- a/docs/zh-CN/docs/advanced/partition/auto-partition.md +++ b/docs/zh-CN/docs/advanced/partition/auto-partition.md @@ -197,3 +197,4 @@ mysql> show partitions from `DAILY_TRADE_VALUE`; - 使用AUTO PARTITION的表,只是分区创建方式上由手动转为了自动。表及其所创建分区的原本使用方法都与非AUTO PARTITION的表或分区相同。 - 为防止意外创建过多分区,我们通过[FE配置项](../../admin-manual/config/fe-config)中的`max_auto_partition_num`控制了一个AUTO PARTITION表最大容纳分区数。如有需要可以调整该值 - 向开启了AUTO PARTITION的表导入数据时,Coordinator发送数据的轮询间隔与普通表有所不同。具体请见[BE配置项](../../admin-manual/config/be-config)中的`olap_table_sink_send_interval_auto_partition_factor`。 +- 在使用[insert-overwrite](../../sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT-OVERWRITE)插入数据时,如果指定了覆写的partition,则AUTO PARTITION表在此过程中表现得如同普通表,不创建新的分区。 diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT-OVERWRITE.md b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT-OVERWRITE.md index f35a65778f..c506029155 100644 --- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT-OVERWRITE.md +++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/INSERT-OVERWRITE.md @@ -68,9 +68,9 @@ INSERT OVERWRITE table table_name 注意: -在当前版本中,会话变量 `enable_insert_strict` 默认为 `true`,如果执行 `INSERT OVERWRITE` 语句时,对于有不符合目标表格式的数据被过滤掉的话会重写目标表失败(比如重写分区时,不满足所有分区条件的数据会被过滤)。 - -INSERT OVERWRITE语句会首先创建一个新表,将需要重写的数据插入到新表中,最后原子性的用新表替换旧表并修改名称。因此,在重写表的过程中,旧表中的数据在重写完毕之前仍然可以正常访问。 +1. 在当前版本中,会话变量 `enable_insert_strict` 默认为 `true`,如果执行 `INSERT OVERWRITE` 语句时,对于有不符合目标表格式的数据被过滤掉的话会重写目标表失败(比如重写分区时,不满足所有分区条件的数据会被过滤)。 +2. 如果INSERT OVERWRITE的目标表是[AUTO-PARTITION表](../../../../advanced/partition/auto-partition),若未指定PARTITION(重写整表),那么可以创建新的分区。如果指定了覆写的PARTITION,那么在此过程中,AUTO PARTITION表表现得如同普通分区表一样,不满足现有分区条件的数据将被过滤,而非创建新的分区。 +3. INSERT OVERWRITE语句会首先创建一个新表,将需要重写的数据插入到新表中,最后原子性的用新表替换旧表并修改名称。因此,在重写表的过程中,旧表中的数据在重写完毕之前仍然可以正常访问。 ### Example @@ -178,5 +178,5 @@ PROPERTIES ( ### Keywords - INSERT OVERWRITE + INSERT OVERWRITE, OVERWRITE diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableAsSelectStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableAsSelectStmt.java index de89b29153..c98fa9414a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableAsSelectStmt.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableAsSelectStmt.java @@ -60,7 +60,7 @@ public class CreateTableAsSelectStmt extends DdlStmt { this.columnNames = columnNames; this.queryStmt = queryStmt; this.insertStmt = new NativeInsertStmt(createTableStmt.getDbTbl(), null, null, - queryStmt, null, columnNames); + queryStmt, null, columnNames, true); } /** diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/NativeInsertStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/NativeInsertStmt.java index 2d9755e317..55c3f77e67 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/NativeInsertStmt.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/NativeInsertStmt.java @@ -160,6 +160,7 @@ public class NativeInsertStmt extends InsertStmt { private InsertType insertType = InsertType.NATIVE_INSERT; boolean hasEmptyTargetColumns = false; + private boolean allowAutoPartition = true; enum InsertType { NATIVE_INSERT("insert_"), @@ -193,13 +194,14 @@ public class NativeInsertStmt extends InsertStmt { // Ctor for CreateTableAsSelectStmt and InsertOverwriteTableStmt public NativeInsertStmt(TableName name, PartitionNames targetPartitionNames, LabelName label, - QueryStmt queryStmt, List planHints, List targetColumnNames) { + QueryStmt queryStmt, List planHints, List targetColumnNames, boolean allowAutoPartition) { super(label, null, null); this.tblName = name; this.targetPartitionNames = targetPartitionNames; this.queryStmt = queryStmt; this.planHints = planHints; this.targetColumnNames = targetColumnNames; + this.allowAutoPartition = allowAutoPartition; this.isValuesOrConstantSelect = (queryStmt instanceof SelectStmt && ((SelectStmt) queryStmt).getTableRefs().isEmpty()); } @@ -1014,6 +1016,9 @@ public class NativeInsertStmt extends InsertStmt { public void complete() throws UserException { if (!isExplain() && targetTable instanceof OlapTable) { ((OlapTableSink) dataSink).complete(analyzer); + if (!allowAutoPartition) { + ((OlapTableSink) dataSink).setAutoPartition(false); + } // add table indexes to transaction state TransactionState txnState = Env.getCurrentGlobalTransactionMgr() .getTransactionState(db.getId(), transactionId); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertExecutor.java index bd16071104..000fd477b3 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertExecutor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertExecutor.java @@ -152,7 +152,8 @@ public class InsertExecutor { /** * finalize sink to complete enough info for sink execution */ - public void finalizeSink(DataSink sink, boolean isPartialUpdate, boolean isFromInsert) { + public void finalizeSink(DataSink sink, boolean isPartialUpdate, boolean isFromInsert, + boolean allowAutoPartition) { if (!(sink instanceof OlapTableSink)) { return; } @@ -170,6 +171,9 @@ public class InsertExecutor { false, isStrictMode); olapTableSink.complete(new Analyzer(Env.getCurrentEnv(), ctx)); + if (!allowAutoPartition) { + olapTableSink.setAutoPartition(false); + } } catch (Exception e) { throw new AnalysisException(e.getMessage(), e); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java index a8dd89f5a9..506637d189 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java @@ -82,6 +82,7 @@ public class InsertIntoTableCommand extends Command implements ForwardWithSync, * When source it's from job scheduler,it will be set. */ private long jobId; + private boolean allowAutoPartition; /** * constructor @@ -90,6 +91,8 @@ public class InsertIntoTableCommand extends Command implements ForwardWithSync, super(PlanType.INSERT_INTO_TABLE_COMMAND); this.logicalQuery = Objects.requireNonNull(logicalQuery, "logicalQuery should not be null"); this.labelName = Objects.requireNonNull(labelName, "labelName should not be null"); + // only insert overwrite will disable it. + this.allowAutoPartition = true; } public void setLabelName(Optional labelName) { @@ -100,6 +103,10 @@ public class InsertIntoTableCommand extends Command implements ForwardWithSync, this.jobId = jobId; } + public void setAllowAutoPartition(boolean allowAutoPartition) { + this.allowAutoPartition = allowAutoPartition; + } + @Override public void run(ConnectContext ctx, StmtExecutor executor) throws Exception { if (!ctx.getSessionVariable().isEnableNereidsDML()) { @@ -160,7 +167,7 @@ public class InsertIntoTableCommand extends Command implements ForwardWithSync, physicalOlapTableSink.getTargetTable(), label, planner); insertExecutor.beginTransaction(); insertExecutor.finalizeSink(sink, physicalOlapTableSink.isPartialUpdate(), - physicalOlapTableSink.getDmlCommandType() == DMLCommandType.INSERT); + physicalOlapTableSink.getDmlCommandType() == DMLCommandType.INSERT, this.allowAutoPartition); } finally { targetTableIf.readUnlock(); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertOverwriteTableCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertOverwriteTableCommand.java index f353b80267..95a36146c7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertOverwriteTableCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertOverwriteTableCommand.java @@ -201,7 +201,10 @@ public class InsertOverwriteTableCommand extends Command implements ForwardWithS sink.isPartialUpdate(), sink.getDMLCommandType(), (LogicalPlan) (sink.child(0))); - new InsertIntoTableCommand(copySink, labelName).run(ctx, executor); + // for overwrite situation, we disable auto create partition. + InsertIntoTableCommand insertCommand = new InsertIntoTableCommand(copySink, labelName); + insertCommand.setAllowAutoPartition(false); + insertCommand.run(ctx, executor); if (ctx.getState().getStateType() == MysqlStateType.ERR) { String errMsg = Strings.emptyToNull(ctx.getState().getErrorMessage()); LOG.warn("InsertInto state error:{}", errMsg); diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java index 923c12df06..f85a621c96 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java @@ -172,6 +172,10 @@ public class OlapTableSink extends DataSink { tDataSink.getOlapTableSink().setLoadId(newLoadId); } + public void setAutoPartition(boolean var) { + tDataSink.getOlapTableSink().getPartition().setEnableAutomaticPartition(var); + } + // must called after tupleDescriptor is computed public void complete(Analyzer analyzer) throws UserException { for (Long partitionId : partitionIds) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java index 3556ae311b..6dab383e06 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java @@ -2522,7 +2522,7 @@ public class StmtExecutor { // after success create table insert data try { parsedStmt = new NativeInsertStmt(tmpTableName, null, new LabelName(iotStmt.getDb(), iotStmt.getLabel()), - iotStmt.getQueryStmt(), iotStmt.getHints(), iotStmt.getCols()); + iotStmt.getQueryStmt(), iotStmt.getHints(), iotStmt.getCols(), true); parsedStmt.setUserInfo(context.getCurrentUserIdentity()); execute(); if (MysqlStateType.ERR.equals(context.getState().getStateType())) { @@ -2595,7 +2595,7 @@ public class StmtExecutor { try { parsedStmt = new NativeInsertStmt(targetTableName, new PartitionNames(true, tempPartitionName), new LabelName(iotStmt.getDb(), iotStmt.getLabel()), iotStmt.getQueryStmt(), - iotStmt.getHints(), iotStmt.getCols()); + iotStmt.getHints(), iotStmt.getCols(), false); parsedStmt.setUserInfo(context.getCurrentUserIdentity()); execute(); if (MysqlStateType.ERR.equals(context.getState().getStateType())) { diff --git a/regression-test/data/partition_p0/auto_partition/test_auto_partition_behavior.out b/regression-test/data/partition_p0/auto_partition/test_auto_partition_behavior.out index ac0ac36cee..d24d4db21d 100644 --- a/regression-test/data/partition_p0/auto_partition/test_auto_partition_behavior.out +++ b/regression-test/data/partition_p0/auto_partition/test_auto_partition_behavior.out @@ -117,3 +117,9 @@ xxX 3 2013-12-12T00:00 2013-12-12T00:00 2020-12-12T00:00 2020-12-12T12:12:12.123456 +-- !sql_overwrite1 -- +Yyy + +-- !sql_overwrite2 -- +Xxx + diff --git a/regression-test/suites/partition_p0/auto_partition/test_auto_partition_behavior.groovy b/regression-test/suites/partition_p0/auto_partition/test_auto_partition_behavior.groovy index 8784c68839..f42fbcee4a 100644 --- a/regression-test/suites/partition_p0/auto_partition/test_auto_partition_behavior.groovy +++ b/regression-test/suites/partition_p0/auto_partition/test_auto_partition_behavior.groovy @@ -153,4 +153,43 @@ suite("test_auto_partition_behavior") { sql """ insert into agg_dt6 values ('2020-12-12', '2020-12-12'), ('2020-12-12', '2020-12-12 12:12:12.123456'), ('2020-12-12', '20121212'), (20131212, 20131212) """ result = sql "show partitions from agg_dt6" assertEquals(result.size(), 5) + + /// insert overwrite + sql "drop table if exists `rewrite`" + sql """ + CREATE TABLE `rewrite` ( + `str` varchar not null + ) ENGINE=OLAP + DUPLICATE KEY(`str`) + COMMENT 'OLAP' + AUTO PARTITION BY LIST (`str`) + ( + PARTITION `p1` values in (("Xxx"), ("Yyy")) + ) + DISTRIBUTED BY HASH(`str`) BUCKETS 10 + PROPERTIES ( + "replication_allocation" = "tag.location.default: 1" + ); + """ + sql """ insert into rewrite values ("Xxx"); """ + // legacy planner + sql " set experimental_enable_nereids_planner=false " + try { + sql """ insert overwrite table rewrite partition(p1) values ("XXX") """ + fail() + } catch (Exception e) { + assertTrue(e.getMessage().contains("Insert has filtered data in strict mode")) + } + sql """ insert overwrite table rewrite partition(p1) values ("Yyy") """ + qt_sql_overwrite1 """ select * from rewrite """ // Yyy + // nereids planner + sql " set experimental_enable_nereids_planner=true " + try { + sql """ insert overwrite table rewrite partition(p1) values ("") """ + fail() + } catch (Exception e) { + assertTrue(e.getMessage().contains("Insert has filtered data in strict mode")) + } + sql """ insert overwrite table rewrite partition(p1) values ("Xxx") """ + qt_sql_overwrite2 """ select * from rewrite """ // Xxx }