diff --git a/docs/documentation/cn/administrator-guide/load-data/routine-load-manual.md b/docs/documentation/cn/administrator-guide/load-data/routine-load-manual.md index c422021498..b22504bb32 100644 --- a/docs/documentation/cn/administrator-guide/load-data/routine-load-manual.md +++ b/docs/documentation/cn/administrator-guide/load-data/routine-load-manual.md @@ -63,7 +63,7 @@ FE 中的 JobScheduler 根据汇报结果，继续生成后续新的 Task，或 ### 创建例行导入任务 -创建例行导入任务的的详细语法可以连接到 Doris 后，执行 `HELP CREATE ROUTINE LOAD;` 查看语法帮助。这里主要详细介绍，创建作业时的注意事项。 +创建例行导入任务的的详细语法可以连接到 Doris 后，执行 `HELP ROUTINE LOAD;` 查看语法帮助。这里主要详细介绍，创建作业时的注意事项。 * columns_mapping diff --git a/docs/documentation/cn/sql-reference/sql-statements/Data Definition/CREATE TABLE.md b/docs/documentation/cn/sql-reference/sql-statements/Data Definition/CREATE TABLE.md index 493d8c3731..9a3752d08d 100644 --- a/docs/documentation/cn/sql-reference/sql-statements/Data Definition/CREATE TABLE.md +++ b/docs/documentation/cn/sql-reference/sql-statements/Data Definition/CREATE TABLE.md @@ -112,12 +112,13 @@ 除AGGREGATE KEY外，其他key_type在建表时，value列不需要指定聚合类型。 4. partition_desc - 1) Range 分区 + partition描述有两种使用方式 + 1) LESS THAN 语法： PARTITION BY RANGE (k1, k2, ...) ( - PARTITION partition_name VALUES LESS THAN MAXVALUE|("value1", "value2", ...) - PARTITION partition_name VALUES LESS THAN MAXVALUE|("value1", "value2", ...) + PARTITION partition_name1 VALUES LESS THAN MAXVALUE|("value1", "value2", ...), + PARTITION partition_name2 VALUES LESS THAN MAXVALUE|("value1", "value2", ...) ... ) 说明： @@ -132,6 +133,17 @@ 注意： 1) 分区一般用于时间维度的数据管理 2) 有数据回溯需求的，可以考虑首个分区为空分区，以便后续增加分区 + 2）Fixed Range + 语法： + PARTITION BY RANGE (k1, k2, k3, ...) + ( + PARTITION partition_name1 VALUES [("k1-lower1", "k2-lower1", "k3-lower1",...), ("k1-upper1", "k2-upper1", "k3-upper1", ...)), + PARTITION partition_name2 VALUES [("k1-lower1-2", "k2-lower1-2", ...), ("k1-upper1-2", MAXVALUE, )) + "k3-upper1-2", ... + ) + 说明： + 1）Fixed Range比LESS THAN相对灵活些，左右区间完全由用户自己确定 + 2）其他与LESS THAN保持同步 5. distribution_desc 1) Hash 分桶 @@ -216,6 +228,7 @@ 3. 创建一个 olap 表，使用 Key Range 分区，使用Hash分桶，默认使用列存， 相同key的记录同时存在，设置初始存储介质和冷却时间 + 1）LESS THAN CREATE TABLE example_db.table_range ( k1 DATE, @@ -244,7 +257,27 @@ [ {"2014-06-01"}, {"2014-12-01"} ) 不在这些分区范围内的数据将视为非法数据被过滤 - + 2) Fixed Range + CREATE TABLE table_range + ( + k1 DATE, + k2 INT, + k3 SMALLINT, + v1 VARCHAR(2048), + v2 DATETIME DEFAULT "2014-02-04 15:36:00" + ) + ENGINE=olap + DUPLICATE KEY(k1, k2, k3) + PARTITION BY RANGE (k1, k2, k3) + ( + PARTITION p1 VALUES [("2014-01-01", "10", "200"), ("2014-01-01", "20", "300")), + PARTITION p2 VALUES [("2014-06-01", "100", "200"), ("2014-07-01", "100", "300")) + ) + DISTRIBUTED BY HASH(k2) BUCKETS 32 + PROPERTIES( + "storage_medium" = "SSD" + ); + 4. 创建一个 mysql 表 CREATE TABLE example_db.table_mysql ( diff --git a/docs/documentation/en/administrator-guide/load-data/routine-load-manual_EN.md b/docs/documentation/en/administrator-guide/load-data/routine-load-manual_EN.md index e6aa10549f..cc339c8219 100644 --- a/docs/documentation/en/administrator-guide/load-data/routine-load-manual_EN.md +++ b/docs/documentation/en/administrator-guide/load-data/routine-load-manual_EN.md @@ -63,7 +63,7 @@ Currently we only support routine imports from Kafka systems. This section detai ### Create routine import tasks -After creating a detailed grammar for routine import tasks, you can connect to Doris and execute `HELP CREATE ROUTINE LOAD'; `Check grammar help. Here is a detailed description of the creation of the job note. +After creating a detailed grammar for routine import tasks, you can connect to Doris and execute `HELP ROUTINE LOAD'; `Check grammar help. Here is a detailed description of the creation of the job note. * columns_mapping diff --git a/docs/documentation/en/sql-reference/sql-statements/Data Definition/CREATE TABLE_EN.md b/docs/documentation/en/sql-reference/sql-statements/Data Definition/CREATE TABLE_EN.md index d64a2f604b..150216421c 100644 --- a/docs/documentation/en/sql-reference/sql-statements/Data Definition/CREATE TABLE_EN.md +++ b/docs/documentation/en/sql-reference/sql-statements/Data Definition/CREATE TABLE_EN.md @@ -139,16 +139,16 @@ Except for AGGREGATE KEY, no need to specify aggregation type for value columns. 4. partition_desc - - 1) Range partition + Partition has two ways to use: + 1) LESS THAN Syntex: ``` PARTITION BY RANGE (k1, k2, ...) ( - PARTITION partition_name VALUES LESS THAN MAXVALUE|("value1", "value2", ...) - PARTITION partition_name VALUES LESS THAN MAXVALUE|("value1", "value2", ...) + PARTITION partition_name1 VALUES LESS THAN MAXVALUE|("value1", "value2", ...), + PARTITION partition_name2 VALUES LESS THAN MAXVALUE|("value1", "value2", ...) ... ) ``` @@ -161,7 +161,23 @@ 3) The range is [closed, open). And the lower bound of first partition is MIN VALUE of specifed column type. 4) NULL values should be save in partition which includes MIN VALUE. 5) Support multi partition columns, the the default partition value is MIN VALUE. - + + 2）Fixed Range + Syntex: + + ``` + PARTITION BY RANGE (k1, k2, k3, ...) + ( + PARTITION partition_name1 VALUES [("k1-lower1", "k2-lower1", "k3-lower1",...), ("k1-upper1", "k2-upper1", "k3-upper1", ...)), + PARTITION partition_name2 VALUES [("k1-lower1-2", "k2-lower1-2", ...), ("k1-upper1-2", MAXVALUE, )) + "k3-upper1-2", ... + ) + ``` + + Explain: + 1）The Fixed Range is more flexible than the LESS THAN, and the left and right intervals are completely determined by the user. + 2）Others are consistent with LESS THAN. + 5. distribution_desc 1) Hash Syntax: @@ -248,7 +264,7 @@ ); 3. Create an olap table, with range partitioned, distributed by hash. - + 1) LESS THAN ``` CREATE TABLE example_db.table_range ( @@ -283,6 +299,27 @@ Data outside these ranges will not be loaded. + 2) Fixed Range + CREATE TABLE table_range + ( + k1 DATE, + k2 INT, + k3 SMALLINT, + v1 VARCHAR(2048), + v2 DATETIME DEFAULT "2014-02-04 15:36:00" + ) + ENGINE=olap + DUPLICATE KEY(k1, k2, k3) + PARTITION BY RANGE (k1, k2, k3) + ( + PARTITION p1 VALUES [("2014-01-01", "10", "200"), ("2014-01-01", "20", "300")), + PARTITION p2 VALUES [("2014-06-01", "100", "200"), ("2014-07-01", "100", "300")) + ) + DISTRIBUTED BY HASH(k2) BUCKETS 32 + PROPERTIES( + "storage_medium" = "SSD" + ); + 4. Create a mysql table ``` diff --git a/fe/src/main/cup/sql_parser.cup b/fe/src/main/cup/sql_parser.cup index 2bdfad742e..156ff902b9 100644 --- a/fe/src/main/cup/sql_parser.cup +++ b/fe/src/main/cup/sql_parser.cup @@ -366,6 +366,8 @@ nonterminal Long opt_field_length; nonterminal KeysDesc opt_keys; nonterminal PartitionKeyDesc partition_key_desc; +nonterminal PartitionKeyDesc fixed_partition_key_desc; +nonterminal List partition_key_list; nonterminal SingleRangePartitionDesc single_range_partition_desc; nonterminal List opt_single_range_partition_desc_list; nonterminal List single_range_partition_desc_list; @@ -1545,6 +1547,11 @@ single_range_partition_desc ::= {: RESULT = new SingleRangePartitionDesc(ifNotExists, partName, desc, properties); :} + | KW_PARTITION opt_if_not_exists:ifNotExists ident:partName KW_VALUES fixed_partition_key_desc:desc + opt_key_value_map:properties + {: + RESULT = new SingleRangePartitionDesc(ifNotExists, partName, desc, properties); + :} ; partition_key_desc ::= @@ -1552,12 +1559,46 @@ partition_key_desc ::= {: RESULT = PartitionKeyDesc.createMaxKeyDesc(); :} - | LPAREN string_list:keys RPAREN + | LPAREN partition_key_list:keys RPAREN {: RESULT = new PartitionKeyDesc(keys); :} ; +partition_key_list ::= + /* empty */ + {: + List l = new ArrayList(); + RESULT = l; + :} + | partition_key_list:l COMMA STRING_LITERAL:item + {: + l.add(new PartitionValue(item)); + RESULT = l; + :} + | partition_key_list:l COMMA KW_MAX_VALUE + {: + l.add(PartitionValue.createMaxValue()); + RESULT = l; + :} + | STRING_LITERAL:item + {: + RESULT = Lists.newArrayList(new PartitionValue(item)); + :} + | KW_MAX_VALUE + {: + RESULT = Lists.newArrayList(PartitionValue.createMaxValue()); + :} + ; + +fixed_partition_key_desc ::= + /* format: [(lower), (upper))*/ + LBRACKET LPAREN partition_key_list:lower RPAREN COMMA LPAREN partition_key_list:upper RPAREN RPAREN + {: + RESULT = new PartitionKeyDesc(lower, upper); + :} + ; + opt_engine ::= {: RESULT = null; :} | KW_ENGINE EQUAL ident:engineName diff --git a/fe/src/main/java/org/apache/doris/analysis/PartitionKeyDesc.java b/fe/src/main/java/org/apache/doris/analysis/PartitionKeyDesc.java index 484262da2a..371372941d 100644 --- a/fe/src/main/java/org/apache/doris/analysis/PartitionKeyDesc.java +++ b/fe/src/main/java/org/apache/doris/analysis/PartitionKeyDesc.java @@ -32,86 +32,122 @@ import java.util.List; // 用于表达在创建表、创建rollup中key range partition中所使用的key信息 // 在知道具体列信息后，可以转成PartitionKey，仅仅在语法解析中短暂的有意义 public class PartitionKeyDesc implements Writable { - // public static final PartitionKeyDesc MAX_VALUE = new PartitionKeyDesc(); - // lower values only be used for restore - private List lowerValues; - private List upperValues; + public enum PartitionRangeType { + INVALID, + LESS_THAN, + FIXED + } + private List lowerValues; + private List upperValues; + private PartitionRangeType partitionType; public static PartitionKeyDesc createMaxKeyDesc() { return new PartitionKeyDesc(); } public PartitionKeyDesc() { - lowerValues = Lists.newArrayList(); - upperValues = Lists.newArrayList(); + partitionType = PartitionRangeType.LESS_THAN; //LESS_THAN is default type. } // used by SQL parser - public PartitionKeyDesc(List upperValues) { - this.lowerValues = Lists.newArrayList(); + public PartitionKeyDesc(List upperValues) { this.upperValues = upperValues; + partitionType = PartitionRangeType.LESS_THAN; } - public PartitionKeyDesc(List lowerValues, List upperValues) { + public PartitionKeyDesc(List lowerValues, List upperValues) { this.lowerValues = lowerValues; this.upperValues = upperValues; + partitionType = PartitionRangeType.FIXED; } - public void setLowerValues(List lowerValues) { + public void setLowerValues(List lowerValues) { this.lowerValues = lowerValues; } - public List getLowerValues() { + public List getLowerValues() { return lowerValues; } - public List getUpperValues() { + public List getUpperValues() { return upperValues; } public boolean isMax() { - return upperValues.isEmpty(); + return lowerValues == null && upperValues == null; + } + + public boolean hasLowerValues() { + return lowerValues != null; + } + + public boolean hasUpperValues() { + return upperValues != null; + } + + public PartitionRangeType getPartitionType () { + return partitionType; } public String toSql() { if (this.isMax()) { return "MAXVALUE"; } - StringBuilder sb = new StringBuilder("("); - Joiner.on(", ").appendTo(sb, Lists.transform(upperValues, new Function() { - @Override - public String apply(String s) { - return "'" + s + "'"; - } - })).append(")"); - return sb.toString(); + + if (upperValues != null) { + StringBuilder sb = new StringBuilder("("); + Joiner.on(", ").appendTo(sb, Lists.transform(upperValues, new Function() { + @Override + public String apply(PartitionValue v) { + return "'" + v.getStringValue() + "'"; + } + })).append(")"); + return sb.toString(); + } else { + return "()"; + } } @Override public void write(DataOutput out) throws IOException { - int count = lowerValues.size(); + int count = lowerValues == null ? 0 : lowerValues.size(); out.writeInt(count); - for (String value : lowerValues) { - Text.writeString(out, value); + if (count > 0) { + for (PartitionValue value : lowerValues) { + Text.writeString(out, value.getStringValue()); + } } - count = upperValues.size(); + count = upperValues == null ? 0: upperValues.size(); out.writeInt(count); - for (String value : upperValues) { - Text.writeString(out, value); + if (count > 0) { + for (PartitionValue value : upperValues) { + Text.writeString(out, value.getStringValue()); + } } + } @Override public void readFields(DataInput in) throws IOException { int count = in.readInt(); for (int i = 0; i < count; i++) { - lowerValues.add(Text.readString(in)); + String v = Text.readString(in); + if (v.equals("MAXVALUE")) { + lowerValues.add(new PartitionValue()); + } else { + lowerValues.add(new PartitionValue(v)); + } } count = in.readInt(); for (int i = 0; i < count; i++) { - upperValues.add(Text.readString(in)); + String v = Text.readString(in); + if (v.equals("MAXVALUE")) { + upperValues.add(new PartitionValue()); + } else { + upperValues.add(new PartitionValue(v)); + } } } } diff --git a/fe/src/main/java/org/apache/doris/analysis/PartitionValue.java b/fe/src/main/java/org/apache/doris/analysis/PartitionValue.java new file mode 100644 index 0000000000..6a2a414624 --- /dev/null +++ b/fe/src/main/java/org/apache/doris/analysis/PartitionValue.java @@ -0,0 +1,62 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.analysis; + +import org.apache.doris.catalog.Type; +import org.apache.doris.common.AnalysisException; + +public class PartitionValue { + private String value; + private boolean isMaxValue; + + public static PartitionValue createMaxValue() { + PartitionValue value = new PartitionValue(); + value.isMaxValue = true; + value.value = null; + return value; + } + + public PartitionValue() { + + } + + public PartitionValue(String value) { + this.isMaxValue = false; + this.value = value; + } + + public LiteralExpr getValue(Type type) throws AnalysisException { + if (isMaxValue) { + return LiteralExpr.createInfinity(type, true); + } else { + return LiteralExpr.create(value, type); + } + } + + public boolean isMax() { + return isMaxValue; + } + + public String getStringValue() { + if (isMaxValue) { + return "MAXVALUE"; + } else { + return value; + } + } +} diff --git a/fe/src/main/java/org/apache/doris/analysis/RangePartitionDesc.java b/fe/src/main/java/org/apache/doris/analysis/RangePartitionDesc.java index 3e539e4da9..b7449e2bfd 100644 --- a/fe/src/main/java/org/apache/doris/analysis/RangePartitionDesc.java +++ b/fe/src/main/java/org/apache/doris/analysis/RangePartitionDesc.java @@ -21,6 +21,7 @@ import org.apache.doris.catalog.Column; import org.apache.doris.catalog.PartitionInfo; import org.apache.doris.catalog.PartitionType; import org.apache.doris.catalog.RangePartitionInfo; +import org.apache.doris.analysis.PartitionKeyDesc.PartitionRangeType; import org.apache.doris.common.AnalysisException; import org.apache.doris.common.DdlException; import org.apache.doris.common.io.Text; @@ -94,6 +95,7 @@ public class RangePartitionDesc extends PartitionDesc { } Set nameSet = Sets.newTreeSet(String.CASE_INSENSITIVE_ORDER); + PartitionRangeType partitionType = PartitionRangeType.INVALID; for (SingleRangePartitionDesc desc : singleRangePartitionDescs) { if (!nameSet.add(desc.getPartitionName())) { throw new AnalysisException("Duplicated partition name: " + desc.getPartitionName()); @@ -104,6 +106,12 @@ public class RangePartitionDesc extends PartitionDesc { if (otherProperties != null) { givenProperties = Maps.newHashMap(otherProperties); } + // check partitionType + if (partitionType == PartitionRangeType.INVALID) { + partitionType = desc.getPartitionKeyDesc().getPartitionType(); + } else if (partitionType != desc.getPartitionKeyDesc().getPartitionType()) { + throw new AnalysisException("You can only use one of these methods to create partitions"); + } desc.analyze(columnDefs.size(), givenProperties); } } diff --git a/fe/src/main/java/org/apache/doris/catalog/PartitionKey.java b/fe/src/main/java/org/apache/doris/catalog/PartitionKey.java index f34633ec1c..a80a180229 100644 --- a/fe/src/main/java/org/apache/doris/catalog/PartitionKey.java +++ b/fe/src/main/java/org/apache/doris/catalog/PartitionKey.java @@ -22,6 +22,7 @@ import org.apache.doris.analysis.IntLiteral; import org.apache.doris.analysis.LargeIntLiteral; import org.apache.doris.analysis.LiteralExpr; import org.apache.doris.analysis.MaxLiteral; +import org.apache.doris.analysis.PartitionValue; import org.apache.doris.common.AnalysisException; import org.apache.doris.common.io.Text; import org.apache.doris.common.io.Writable; @@ -62,13 +63,13 @@ public class PartitionKey implements Comparable, Writable { return partitionKey; } - public static PartitionKey createPartitionKey(List keys, List columns) + public static PartitionKey createPartitionKey(List keys, List columns) throws AnalysisException { PartitionKey partitionKey = new PartitionKey(); Preconditions.checkArgument(keys.size() <= columns.size()); int i; for (i = 0; i < keys.size(); ++i) { - partitionKey.keys.add(LiteralExpr.create(keys.get(i), + partitionKey.keys.add(keys.get(i).getValue( Type.fromPrimitiveType(columns.get(i).getDataType()))); partitionKey.types.add(columns.get(i).getDataType()); } @@ -127,6 +128,29 @@ public class PartitionKey implements Comparable, Writable { return true; } + public static int compareLiteralExpr(LiteralExpr key1, LiteralExpr key2) { + int ret = 0; + if (key1 instanceof MaxLiteral || key2 instanceof MaxLiteral) { + ret = key1.compareLiteral(key2); + } else { + final Type destType = Type.getAssignmentCompatibleType(key1.getType(), key2.getType(), false); + try { + LiteralExpr newKey = key1; + if (key1.getType() != destType) { + newKey = (LiteralExpr) key1.castTo(destType); + } + LiteralExpr newOtherKey = key2; + if (key2.getType() != destType) { + newOtherKey = (LiteralExpr) key2.castTo(destType); + } + ret = newKey.compareLiteral(newOtherKey); + } catch (AnalysisException e) { + throw new RuntimeException("Cast error in partition"); + } + } + return ret; + } + // compare with other PartitionKey. used for partition prune @Override public int compareTo(PartitionKey other) { @@ -134,27 +158,7 @@ public class PartitionKey implements Comparable, Writable { int other_key_len = other.keys.size(); int min_len = Math.min(this_key_len, other_key_len); for (int i = 0; i < min_len; ++i) { - final LiteralExpr oldKey = this.getKeys().get(i); - final LiteralExpr otherOldKey = other.getKeys().get(i); - int ret = 0; - if (oldKey instanceof MaxLiteral || otherOldKey instanceof MaxLiteral) { - ret = oldKey.compareLiteral(otherOldKey); - } else { - final Type destType = Type.getAssignmentCompatibleType(oldKey.getType(), otherOldKey.getType(), false); - try { - LiteralExpr newKey = oldKey; - if (oldKey.getType() != destType) { - newKey = (LiteralExpr) oldKey.castTo(destType); - } - LiteralExpr newOtherKey = otherOldKey; - if (otherOldKey.getType() != destType) { - newOtherKey = (LiteralExpr) otherOldKey.castTo(destType); - } - ret = newKey.compareLiteral(newOtherKey); - } catch (AnalysisException e) { - throw new RuntimeException("Cast error in partition"); - } - } + int ret = compareLiteralExpr(this.getKeys().get(i), other.getKeys().get(i)); if (0 != ret) { return ret; } diff --git a/fe/src/main/java/org/apache/doris/catalog/RangePartitionInfo.java b/fe/src/main/java/org/apache/doris/catalog/RangePartitionInfo.java index 404ed8c84e..32ca86372a 100644 --- a/fe/src/main/java/org/apache/doris/catalog/RangePartitionInfo.java +++ b/fe/src/main/java/org/apache/doris/catalog/RangePartitionInfo.java @@ -18,6 +18,7 @@ package org.apache.doris.catalog; import org.apache.doris.analysis.PartitionKeyDesc; +import org.apache.doris.analysis.PartitionValue; import org.apache.doris.analysis.SingleRangePartitionDesc; import org.apache.doris.common.AnalysisException; import org.apache.doris.common.DdlException; @@ -122,7 +123,7 @@ public class RangePartitionInfo extends PartitionInfo { PartitionKey upperKey = nextRange.upperEndpoint(); if (upperKey.compareTo(singlePartitionKey) >= 0) { PartitionKey lowKey = null; - if (!partKeyDesc.getLowerValues().isEmpty()) { + if (partKeyDesc.hasLowerValues()) { lowKey = PartitionKey.createPartitionKey(partKeyDesc.getLowerValues(), partitionColumns); } else { if (lastRange == null) { @@ -131,7 +132,10 @@ public class RangePartitionInfo extends PartitionInfo { lowKey = lastRange.upperEndpoint(); } } - + // check: [left, right), error if left equal right + if (lowKey.compareTo(singlePartitionKey) >= 0) { + throw new IllegalArgumentException("The right value must be more than the left value"); + } newRange = Range.closedOpen(lowKey, singlePartitionKey); // check if range intersected @@ -143,7 +147,7 @@ public class RangePartitionInfo extends PartitionInfo { if (newRange == null) { PartitionKey lowKey = null; - if (!partKeyDesc.getLowerValues().isEmpty()) { + if (partKeyDesc.hasLowerValues()) { lowKey = PartitionKey.createPartitionKey(partKeyDesc.getLowerValues(), partitionColumns); } else { if (lastRange == null) { @@ -153,8 +157,16 @@ public class RangePartitionInfo extends PartitionInfo { lowKey = lastRange.upperEndpoint(); } } - + // check: [left, right), error if left equal right + if (lowKey.compareTo(singlePartitionKey) >= 0) { + throw new IllegalArgumentException("The right value must be more than the left value"); + } newRange = Range.closedOpen(lowKey, singlePartitionKey); + + // check if range intersected. The first Partition if lastRange == null + if (lastRange != null) { + checkRangeIntersect(newRange, lastRange); + } } } catch (AnalysisException e) { throw new DdlException("Invalid range value format： " + e.getMessage()); @@ -181,7 +193,7 @@ public class RangePartitionInfo extends PartitionInfo { idToRange.put(partitionId, range); } catch (IllegalArgumentException e) { // Range.closedOpen may throw this if (lower > upper) - throw new DdlException("Invalid key range", e); + throw new DdlException("Invalid key range: " + e.getMessage()); } idToDataProperty.put(partitionId, desc.getPartitionDataProperty()); idToReplicationNum.put(partitionId, desc.getReplicationNum()); @@ -298,10 +310,10 @@ public class RangePartitionInfo extends PartitionInfo { public SingleRangePartitionDesc toSingleRangePartitionDesc(long partitionId, String partitionName, Map properties) { Range range = idToRange.get(partitionId); - List upperValues = Lists.newArrayList(); - List lowerValues = Lists.newArrayList(); + List upperValues = Lists.newArrayList(); + List lowerValues = Lists.newArrayList(); // FIXME(cmy): check here(getStringValue) - lowerValues.add(range.lowerEndpoint().getKeys().get(0).getStringValue()); + lowerValues.add(new PartitionValue(range.lowerEndpoint().getKeys().get(0).getStringValue())); PartitionKey upperKey = range.upperEndpoint(); PartitionKeyDesc keyDesc = null; @@ -309,7 +321,7 @@ public class RangePartitionInfo extends PartitionInfo { keyDesc = PartitionKeyDesc.createMaxKeyDesc(); keyDesc.setLowerValues(lowerValues); } else { - upperValues.add(range.upperEndpoint().getKeys().get(0).getStringValue()); + upperValues.add(new PartitionValue(range.upperEndpoint().getKeys().get(0).getStringValue())); keyDesc = new PartitionKeyDesc(lowerValues, upperValues); } diff --git a/fe/src/main/java/org/apache/doris/common/util/KuduUtil.java b/fe/src/main/java/org/apache/doris/common/util/KuduUtil.java index 7652db26cd..f488bcb5af 100644 --- a/fe/src/main/java/org/apache/doris/common/util/KuduUtil.java +++ b/fe/src/main/java/org/apache/doris/common/util/KuduUtil.java @@ -26,8 +26,8 @@ import org.apache.doris.analysis.RangePartitionDesc; import org.apache.doris.analysis.SingleRangePartitionDesc; import org.apache.doris.catalog.Column; import org.apache.doris.catalog.KeysType; -import org.apache.doris.catalog.PrimitiveType; import org.apache.doris.catalog.KuduPartition.KuduRange; +import org.apache.doris.catalog.PrimitiveType; import org.apache.doris.common.AnalysisException; import org.apache.doris.common.Config; import org.apache.doris.common.DdlException; @@ -35,22 +35,22 @@ import org.apache.doris.common.ErrorCode; import org.apache.doris.common.ErrorReport; import org.apache.doris.common.FeNameFormat; -import java.nio.charset.Charset; -import java.util.List; -import java.util.Set; +import com.google.common.base.Preconditions; +import com.google.common.collect.Lists; +import com.google.common.collect.Sets; import org.apache.kudu.ColumnSchema; -import org.apache.kudu.Type; import org.apache.kudu.Schema; +import org.apache.kudu.Type; import org.apache.kudu.client.CreateTableOptions; import org.apache.kudu.client.KuduClient; import org.apache.kudu.client.KuduClient.KuduClientBuilder; import org.apache.kudu.client.PartialRow; import org.apache.kudu.client.RangePartitionBound; -import com.google.common.base.Preconditions; -import com.google.common.collect.Lists; -import com.google.common.collect.Sets; +import java.nio.charset.Charset; +import java.util.List; +import java.util.Set; public class KuduUtil { @@ -105,8 +105,8 @@ public class KuduUtil { PartialRow lower = kuduSchema.newPartialRow(); PartialRow upper = kuduSchema.newPartialRow(); - String lowerValue = lastDesc != null ? lastDesc.getPartitionKeyDesc().getUpperValues().get(0) : ""; - String upperValue = single.getPartitionKeyDesc().getUpperValues().get(0); + String lowerValue = lastDesc != null ? lastDesc.getPartitionKeyDesc().getUpperValues().get(0).getStringValue() : ""; + String upperValue = single.getPartitionKeyDesc().getUpperValues().get(0).getStringValue(); switch (kuduType) { case INT8: diff --git a/fe/src/main/java/org/apache/doris/external/EsIndexState.java b/fe/src/main/java/org/apache/doris/external/EsIndexState.java index c67e7cafa4..5dca5db384 100644 --- a/fe/src/main/java/org/apache/doris/external/EsIndexState.java +++ b/fe/src/main/java/org/apache/doris/external/EsIndexState.java @@ -17,27 +17,27 @@ package org.apache.doris.external; -import java.util.List; -import java.util.Map; -import java.util.Random; -import java.util.stream.Collectors; -import java.util.stream.Stream; - -import org.apache.logging.log4j.LogManager; -import org.apache.logging.log4j.Logger; -import org.json.JSONArray; -import org.json.JSONObject; - import org.apache.doris.analysis.PartitionKeyDesc; +import org.apache.doris.analysis.PartitionValue; import org.apache.doris.analysis.SingleRangePartitionDesc; import org.apache.doris.catalog.PartitionInfo; import org.apache.doris.catalog.PartitionKey; import org.apache.doris.catalog.RangePartitionInfo; import org.apache.doris.common.AnalysisException; import org.apache.doris.thrift.TNetworkAddress; + import com.google.common.collect.Lists; import com.google.common.collect.Maps; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; +import org.json.JSONArray; +import org.json.JSONObject; + +import java.util.List; +import java.util.Map; +import java.util.Random; + public class EsIndexState { private static final Logger LOG = LogManager.getLogger(EsIndexState.class); @@ -112,7 +112,7 @@ public class EsIndexState { partitionSetting == null ? "" : partitionSetting.toString()); if (partitionSetting != null && partitionSetting.has("upperbound")) { String upperBound = partitionSetting.getString("upperbound"); - List upperValues = Stream.of(upperBound).collect(Collectors.toList()); + List upperValues = Lists.newArrayList(new PartitionValue(upperBound)); PartitionKeyDesc partitionKeyDesc = new PartitionKeyDesc(upperValues); // use index name as partition name SingleRangePartitionDesc desc = new SingleRangePartitionDesc(false, diff --git a/fe/src/main/java/org/apache/doris/planner/PartitionColumnFilter.java b/fe/src/main/java/org/apache/doris/planner/PartitionColumnFilter.java index b9097b92ce..32f762283b 100644 --- a/fe/src/main/java/org/apache/doris/planner/PartitionColumnFilter.java +++ b/fe/src/main/java/org/apache/doris/planner/PartitionColumnFilter.java @@ -19,6 +19,7 @@ package org.apache.doris.planner; import org.apache.doris.analysis.InPredicate; import org.apache.doris.analysis.LiteralExpr; +import org.apache.doris.analysis.PartitionValue; import org.apache.doris.catalog.Column; import org.apache.doris.catalog.PartitionKey; import org.apache.doris.common.AnalysisException; @@ -98,8 +99,10 @@ public class PartitionColumnFilter { PartitionKey upperKey = null; // cmy mod, catch AnalysisException try { - lowerKey = PartitionKey.createPartitionKey(Lists.newArrayList(lowerBound.getStringValue()), columns); - upperKey = PartitionKey.createPartitionKey(Lists.newArrayList(upperBound.getStringValue()), columns); + lowerKey = PartitionKey.createPartitionKey( + Lists.newArrayList(new PartitionValue(lowerBound.getStringValue())), columns); + upperKey = PartitionKey.createPartitionKey( + Lists.newArrayList(new PartitionValue(upperBound.getStringValue())), columns); } catch (AnalysisException e) { LOG.warn(e.getMessage()); return null; diff --git a/fe/src/test/java/org/apache/doris/analysis/PartitionKeyDescTest.java b/fe/src/test/java/org/apache/doris/analysis/PartitionKeyDescTest.java index 66057644cb..b694de8a48 100644 --- a/fe/src/test/java/org/apache/doris/analysis/PartitionKeyDescTest.java +++ b/fe/src/test/java/org/apache/doris/analysis/PartitionKeyDescTest.java @@ -25,12 +25,12 @@ import org.junit.Test; import java.util.List; public class PartitionKeyDescTest { - private List values; + private List values; // value of key is ["1", "abc"] @Before public void setUp() { - values = Lists.newArrayList("1", "abc"); + values = Lists.newArrayList(new PartitionValue("1"), new PartitionValue("abc")); } @Test @@ -45,7 +45,7 @@ public class PartitionKeyDescTest { public void testMax() { PartitionKeyDesc desc = PartitionKeyDesc.createMaxKeyDesc(); - Assert.assertTrue(desc.getUpperValues().isEmpty()); + Assert.assertNull(desc.getUpperValues()); Assert.assertEquals("MAXVALUE", desc.toSql()); } -} \ No newline at end of file +} diff --git a/fe/src/test/java/org/apache/doris/backup/AlterClauseRWTest.java b/fe/src/test/java/org/apache/doris/backup/AlterClauseRWTest.java index 206fa0dc9a..8030b07407 100644 --- a/fe/src/test/java/org/apache/doris/backup/AlterClauseRWTest.java +++ b/fe/src/test/java/org/apache/doris/backup/AlterClauseRWTest.java @@ -23,6 +23,7 @@ import org.apache.doris.analysis.AlterClause; import org.apache.doris.analysis.AlterTableStmt; import org.apache.doris.analysis.DistributionDesc; import org.apache.doris.analysis.PartitionKeyDesc; +import org.apache.doris.analysis.PartitionValue; import org.apache.doris.analysis.RandomDistributionDesc; import org.apache.doris.analysis.SingleRangePartitionDesc; import org.apache.doris.analysis.TableName; @@ -64,7 +65,7 @@ public class AlterClauseRWTest { // add partition clause String partititionName = "p1"; - List values = Lists.newArrayList("100"); + List values = Lists.newArrayList(new PartitionValue("100")); PartitionKeyDesc keyDesc = new PartitionKeyDesc(values); Map properties = Maps.newHashMap(); SingleRangePartitionDesc partitionDesc = new SingleRangePartitionDesc(false, partititionName, keyDesc, diff --git a/fe/src/test/java/org/apache/doris/backup/CatalogMocker.java b/fe/src/test/java/org/apache/doris/backup/CatalogMocker.java index 7de0f64d72..57309e6805 100644 --- a/fe/src/test/java/org/apache/doris/backup/CatalogMocker.java +++ b/fe/src/test/java/org/apache/doris/backup/CatalogMocker.java @@ -19,6 +19,7 @@ package org.apache.doris.backup; import org.apache.doris.alter.RollupHandler; import org.apache.doris.alter.SchemaChangeHandler; +import org.apache.doris.analysis.PartitionValue; import org.apache.doris.catalog.AggregateType; import org.apache.doris.catalog.Catalog; import org.apache.doris.catalog.Column; @@ -282,16 +283,16 @@ public class CatalogMocker { PartitionKey rangeP1Lower = PartitionKey.createInfinityPartitionKey(Lists.newArrayList(TEST_TBL_BASE_SCHEMA.get(0)), false); PartitionKey rangeP1Upper = - PartitionKey.createPartitionKey(Lists.newArrayList("10"), + PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("10")), Lists.newArrayList(TEST_TBL_BASE_SCHEMA.get(0))); Range rangeP1 = Range.closedOpen(rangeP1Lower, rangeP1Upper); rangePartitionInfo.setRange(TEST_PARTITION1_ID, rangeP1); PartitionKey rangeP2Lower = - PartitionKey.createPartitionKey(Lists.newArrayList("10"), + PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("10")), Lists.newArrayList(TEST_TBL_BASE_SCHEMA.get(0))); PartitionKey rangeP2Upper = - PartitionKey.createPartitionKey(Lists.newArrayList("20"), + PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("20")), Lists.newArrayList(TEST_TBL_BASE_SCHEMA.get(0))); Range rangeP2 = Range.closedOpen(rangeP2Lower, rangeP2Upper); rangePartitionInfo.setRange(TEST_PARTITION2_ID, rangeP2); diff --git a/fe/src/test/java/org/apache/doris/catalog/CatalogTestUtil.java b/fe/src/test/java/org/apache/doris/catalog/CatalogTestUtil.java index 648b898971..172f860156 100644 --- a/fe/src/test/java/org/apache/doris/catalog/CatalogTestUtil.java +++ b/fe/src/test/java/org/apache/doris/catalog/CatalogTestUtil.java @@ -18,6 +18,7 @@ package org.apache.doris.catalog; import org.apache.doris.analysis.PartitionKeyDesc; +import org.apache.doris.analysis.PartitionValue; import org.apache.doris.analysis.SingleRangePartitionDesc; import org.apache.doris.catalog.MaterializedIndex.IndexExtState; import org.apache.doris.catalog.MaterializedIndex.IndexState; @@ -248,7 +249,7 @@ public class CatalogTestUtil { singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", new PartitionKeyDesc(Lists - .newArrayList("100")), + .newArrayList(new PartitionValue("100"))), null)); RangePartitionInfo partitionInfo = new RangePartitionInfo(partitionColumns); @@ -280,7 +281,7 @@ public class CatalogTestUtil { singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", new PartitionKeyDesc(Lists - .newArrayList("100")), + .newArrayList(new PartitionValue("100"))), null)); SinglePartitionInfo partitionInfo = new SinglePartitionInfo(); diff --git a/fe/src/test/java/org/apache/doris/catalog/PartitionKeyTest.java b/fe/src/test/java/org/apache/doris/catalog/PartitionKeyTest.java index d6f2f2dc91..b7c7cd56e8 100644 --- a/fe/src/test/java/org/apache/doris/catalog/PartitionKeyTest.java +++ b/fe/src/test/java/org/apache/doris/catalog/PartitionKeyTest.java @@ -17,6 +17,7 @@ package org.apache.doris.catalog; +import org.apache.doris.analysis.PartitionValue; import org.apache.doris.common.AnalysisException; import java.io.DataInputStream; @@ -78,77 +79,80 @@ public class PartitionKeyTest { PartitionKey pk2; // case1 - pk1 = PartitionKey.createPartitionKey(Arrays.asList("127", "32767"), + pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("32767")), Arrays.asList(tinyInt, smallInt)); pk2 = PartitionKey.createInfinityPartitionKey(Arrays.asList(tinyInt, smallInt), true); Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1); // case2 - pk1 = PartitionKey.createPartitionKey(Arrays.asList("127"), + pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127")), Arrays.asList(tinyInt, smallInt)); - pk2 = PartitionKey.createPartitionKey(Arrays.asList("127", "-32768"), + pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("-32768")), Arrays.asList(tinyInt, smallInt)); Assert.assertTrue(pk1.equals(pk2) && pk1.compareTo(pk2) == 0); // case3 - pk1 = PartitionKey.createPartitionKey(Arrays.asList("127"), + pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127")), Arrays.asList(int32, bigInt)); - pk2 = PartitionKey.createPartitionKey(Arrays.asList("128", "-32768"), + pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("128"), new PartitionValue("-32768")), Arrays.asList(int32, bigInt)); Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1); // case4 - pk1 = PartitionKey.createPartitionKey(Arrays.asList("127", "12345"), + pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("12345")), Arrays.asList(largeInt, bigInt)); - pk2 = PartitionKey.createPartitionKey(Arrays.asList("127", "12346"), + pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("12346")), Arrays.asList(largeInt, bigInt)); Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1); // case5 - pk1 = PartitionKey.createPartitionKey(Arrays.asList("2014-12-12", "2014-12-12 10:00:00"), + pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("2014-12-12"), new PartitionValue("2014-12-12 10:00:00")), Arrays.asList(date, datetime)); - pk2 = PartitionKey.createPartitionKey(Arrays.asList("2014-12-12", "2014-12-12 10:00:01"), + pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("2014-12-12"), new PartitionValue("2014-12-12 10:00:01")), Arrays.asList(date, datetime)); Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1); // case6 - pk1 = PartitionKey.createPartitionKey(Arrays.asList("-128"), + pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("-128")), Arrays.asList(tinyInt, smallInt)); pk2 = PartitionKey.createInfinityPartitionKey(Arrays.asList(tinyInt, smallInt), false); Assert.assertTrue(pk1.equals(pk2) && pk1.compareTo(pk2) == 0); // case7 - pk1 = PartitionKey.createPartitionKey(Arrays.asList("127"), + pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127")), Arrays.asList(tinyInt, smallInt)); pk2 = PartitionKey.createInfinityPartitionKey(Arrays.asList(tinyInt, smallInt), true); Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1); // case7 - pk1 = PartitionKey.createPartitionKey(Arrays.asList("127", "32767"), + pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("32767")), Arrays.asList(tinyInt, smallInt)); pk2 = PartitionKey.createInfinityPartitionKey(Arrays.asList(tinyInt, smallInt), true); Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1); // case8 - pk1 = PartitionKey.createPartitionKey(Arrays.asList("127", "32767", "2147483647", "9223372036854775807", - "170141183460469231731687303715884105727", - "9999-12-31", "9999-12-31 23:59:59"), - allColumns); + pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("32767"), + new PartitionValue("2147483647"), new PartitionValue("9223372036854775807"), + new PartitionValue("170141183460469231731687303715884105727"), + new PartitionValue("9999-12-31"), new PartitionValue("9999-12-31 23:59:59")), + allColumns); pk2 = PartitionKey.createInfinityPartitionKey(allColumns, true); Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1); // case9 - pk1 = PartitionKey.createPartitionKey(Arrays.asList("-128", "-32768", "-2147483648", "-9223372036854775808", - "-170141183460469231731687303715884105728", - "1900-01-01", "1900-01-01 00:00:00"), - allColumns); + pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("-128"), new PartitionValue("-32768"), + new PartitionValue("-2147483648"), new PartitionValue("-9223372036854775808"), + new PartitionValue("-170141183460469231731687303715884105728"), + new PartitionValue("1900-01-01"), new PartitionValue("1900-01-01 00:00:00")), + allColumns); pk2 = PartitionKey.createInfinityPartitionKey(allColumns, false); Assert.assertTrue(pk1.equals(pk2) && pk1.compareTo(pk2) == 0); // case10 - pk1 = PartitionKey.createPartitionKey(Arrays.asList("-128", "-32768", "0", "-9223372036854775808", - "0", "1970-01-01", "1970-01-01 00:00:00"), - allColumns); + pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("-128"), new PartitionValue("-32768"), + new PartitionValue("0"), new PartitionValue("-9223372036854775808"), + new PartitionValue("0"), new PartitionValue("1970-01-01"), new PartitionValue("1970-01-01 00:00:00")), + allColumns); pk2 = PartitionKey.createInfinityPartitionKey(allColumns, false); Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == 1); } @@ -169,19 +173,19 @@ public class PartitionKeyTest { PartitionKey keyEmpty = new PartitionKey(); keyEmpty.write(dos); - List keys = new ArrayList(); + List keys = new ArrayList(); List columns = new ArrayList(); - keys.add("100"); + keys.add(new PartitionValue("100")); columns.add(new Column("column2", ScalarType.createType(PrimitiveType.TINYINT), true, null, "", "")); - keys.add("101"); + keys.add(new PartitionValue("101")); columns.add(new Column("column3", ScalarType.createType(PrimitiveType.SMALLINT), true, null, "", "")); - keys.add("102"); + keys.add(new PartitionValue("102")); columns.add(new Column("column4", ScalarType.createType(PrimitiveType.INT), true, null, "", "")); - keys.add("103"); + keys.add(new PartitionValue("103")); columns.add(new Column("column5", ScalarType.createType(PrimitiveType.BIGINT), true, null, "", "")); - keys.add("2014-12-26"); + keys.add(new PartitionValue("2014-12-26")); columns.add(new Column("column10", ScalarType.createType(PrimitiveType.DATE), true, null, "", "")); - keys.add("2014-12-27 11:12:13"); + keys.add(new PartitionValue("2014-12-27 11:12:13")); columns.add(new Column("column11", ScalarType.createType(PrimitiveType.DATETIME), true, null, "", "")); PartitionKey key = PartitionKey.createPartitionKey(keys, columns); diff --git a/fe/src/test/java/org/apache/doris/catalog/RangePartitionInfoTest.java b/fe/src/test/java/org/apache/doris/catalog/RangePartitionInfoTest.java index dd98ac73b2..0aa9632b3d 100644 --- a/fe/src/test/java/org/apache/doris/catalog/RangePartitionInfoTest.java +++ b/fe/src/test/java/org/apache/doris/catalog/RangePartitionInfoTest.java @@ -18,6 +18,8 @@ package org.apache.doris.catalog; import org.apache.doris.analysis.PartitionKeyDesc; +import org.apache.doris.analysis.PartitionKeyDesc.PartitionRangeType; +import org.apache.doris.analysis.PartitionValue; import org.apache.doris.analysis.SingleRangePartitionDesc; import org.apache.doris.common.AnalysisException; import org.apache.doris.common.DdlException; @@ -27,6 +29,7 @@ import com.google.common.collect.Lists; import org.junit.Before; import org.junit.Test; +import java.util.ArrayList; import java.util.LinkedList; import java.util.List; @@ -49,8 +52,8 @@ public class RangePartitionInfoTest { partitionColumns.add(k1); singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", - new PartitionKeyDesc(Lists .newArrayList("-128")), - null)); + new PartitionKeyDesc(Lists .newArrayList(new PartitionValue("-128"))), + null)); partitionInfo = new RangePartitionInfo(partitionColumns); @@ -66,8 +69,8 @@ public class RangePartitionInfoTest { partitionColumns.add(k1); singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", - new PartitionKeyDesc(Lists.newArrayList("-32768")), - null)); + new PartitionKeyDesc(Lists.newArrayList(new PartitionValue("-32768"))), + null)); partitionInfo = new RangePartitionInfo(partitionColumns); for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) { @@ -82,9 +85,8 @@ public class RangePartitionInfoTest { partitionColumns.add(k1); singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", - new PartitionKeyDesc(Lists - .newArrayList("-2147483648")), - null)); + new PartitionKeyDesc(Lists.newArrayList(new PartitionValue("-2147483648"))), + null)); partitionInfo = new RangePartitionInfo(partitionColumns); for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) { @@ -99,13 +101,13 @@ public class RangePartitionInfoTest { partitionColumns.add(k1); singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", new PartitionKeyDesc(Lists - .newArrayList("-9223372036854775808")), null)); + .newArrayList(new PartitionValue("-9223372036854775808"))), null)); singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p2", new PartitionKeyDesc(Lists - .newArrayList("-9223372036854775806")), null)); + .newArrayList(new PartitionValue("-9223372036854775806"))), null)); singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p3", new PartitionKeyDesc(Lists - .newArrayList("0")), null)); + .newArrayList(new PartitionValue("0"))), null)); singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p4", new PartitionKeyDesc(Lists - .newArrayList("9223372036854775806")), null)); + .newArrayList(new PartitionValue("9223372036854775806"))), null)); partitionInfo = new RangePartitionInfo(partitionColumns); @@ -121,13 +123,13 @@ public class RangePartitionInfoTest { partitionColumns.add(k1); singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", new PartitionKeyDesc(Lists - .newArrayList("-9223372036854775806")), null)); + .newArrayList(new PartitionValue("-9223372036854775806"))), null)); singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p2", new PartitionKeyDesc(Lists - .newArrayList("-9223372036854775805")), null)); + .newArrayList(new PartitionValue("-9223372036854775805"))), null)); singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p3", new PartitionKeyDesc(Lists - .newArrayList("0")), null)); + .newArrayList(new PartitionValue("0"))), null)); singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p4", new PartitionKeyDesc(Lists - .newArrayList("9223372036854775806")), null)); + .newArrayList(new PartitionValue("9223372036854775806"))), null)); partitionInfo = new RangePartitionInfo(partitionColumns); @@ -137,4 +139,209 @@ public class RangePartitionInfoTest { } } + /** + * PARTITION BY RANGE(`k1`, `k2`) ( + * PARTITION p0 VALUES [("20190101", "100"),("20190101", "200")), + * PARTITION p1 VALUES [("20190105", "10"),("20190107", "10")), + * PARTITION p2 VALUES [("20181231", "10"),("20190101", "100")), + * PARTITION p3 VALUES [("20190105", "100"),("20190120", MAXVALUE)) + * ) + */ + @Test + public void testFixedRange() throws DdlException, AnalysisException { + //add columns + int columns = 2; + Column k1 = new Column("k1", new ScalarType(PrimitiveType.INT), true, null, "", ""); + Column k2 = new Column("k2", new ScalarType(PrimitiveType.BIGINT), true, null, "", ""); + partitionColumns.add(k1); + partitionColumns.add(k2); + + //add RangePartitionDescs + PartitionKeyDesc p1 = new PartitionKeyDesc( + Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("100")), + Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("200"))); + PartitionKeyDesc p2 = new PartitionKeyDesc( + Lists.newArrayList(new PartitionValue("20190105"), new PartitionValue("10")), + Lists.newArrayList(new PartitionValue("20190107"), new PartitionValue("10"))); + PartitionKeyDesc p3 = new PartitionKeyDesc( + Lists.newArrayList(new PartitionValue("20181231"), new PartitionValue("10")), + Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("100"))); + PartitionKeyDesc p4 = new PartitionKeyDesc( + Lists.newArrayList(new PartitionValue("20190105"), new PartitionValue("100")), + Lists.newArrayList(new PartitionValue("20190120"), PartitionValue.createMaxValue())); + + singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", p1, null)); + singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p2", p2, null)); + singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p3", p3, null)); + singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p4", p4, null)); + + partitionInfo = new RangePartitionInfo(partitionColumns); + + for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) { + singleRangePartitionDesc.analyze(columns, null); + partitionInfo.handleNewSinglePartitionDesc(singleRangePartitionDesc, 20000L); + } + } + + /** + * 失败用例 less than && fixed + * partition by range(k1,k2,k3) ( + * partition p1 values less than("2019-02-01", "100", "200"), + * partition p2 values [("2020-02-01", "100", "200"), (MAXVALUE)), + * partition p3 values less than("2021-02-01") + * ) + */ + @Test(expected = AnalysisException.class) + public void testFixedRange1() throws DdlException, AnalysisException { + //add columns + Column k1 = new Column("k1", new ScalarType(PrimitiveType.DATE), true, null, "", ""); + Column k2 = new Column("k2", new ScalarType(PrimitiveType.INT), true, null, "", ""); + Column k3 = new Column("k3", new ScalarType(PrimitiveType.INT), true, null, "", ""); + partitionColumns.add(k1); + partitionColumns.add(k2); + partitionColumns.add(k3); + + //add RangePartitionDescs + PartitionKeyDesc p1 = new PartitionKeyDesc( + Lists.newArrayList(new PartitionValue("2019-02-01"), new PartitionValue("100"), new PartitionValue("200"))); + PartitionKeyDesc p2 = new PartitionKeyDesc( + Lists.newArrayList(new PartitionValue("2020-02-01"), new PartitionValue("100"), new PartitionValue("200")), + Lists.newArrayList(PartitionValue.createMaxValue())); + PartitionKeyDesc p3 = new PartitionKeyDesc( + Lists.newArrayList(new PartitionValue("2021-02-01"))); + + singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", p1, null)); + singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p2", p2, null)); + singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p3", p3, null)); + partitionInfo = new RangePartitionInfo(partitionColumns); + PartitionRangeType partitionType = PartitionRangeType.INVALID; + for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) { + // check partitionType + if (partitionType == PartitionRangeType.INVALID) { + partitionType = singleRangePartitionDesc.getPartitionKeyDesc().getPartitionType(); + } else if (partitionType != singleRangePartitionDesc.getPartitionKeyDesc().getPartitionType()) { + throw new AnalysisException("You can only use one of these methods to create partitions"); + } + singleRangePartitionDesc.analyze(partitionColumns.size(), null); + partitionInfo.handleNewSinglePartitionDesc(singleRangePartitionDesc, 20000L); + } + } + + /** + * PARTITION BY RANGE(`k1`, `k2`) ( + * PARTITION p1 VALUES [(), ("20190301", "400")) + * ) + */ + @Test + public void testFixedRange2() throws DdlException, AnalysisException { + //add columns + int columns = 2; + Column k1 = new Column("k1", new ScalarType(PrimitiveType.INT), true, null, "", ""); + Column k2 = new Column("k2", new ScalarType(PrimitiveType.BIGINT), true, null, "", ""); + partitionColumns.add(k1); + partitionColumns.add(k2); + + //add RangePartitionDescs + PartitionKeyDesc p1 = new PartitionKeyDesc(new ArrayList<>(), + Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("200"))); + + singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", p1, null)); + + partitionInfo = new RangePartitionInfo(partitionColumns); + + for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) { + singleRangePartitionDesc.analyze(columns, null); + partitionInfo.handleNewSinglePartitionDesc(singleRangePartitionDesc, 20000L); + } + } + + /** + * 失败用例 + * PARTITION BY RANGE(`k1`, `k2`) ( + * PARTITION p1 VALUES [("20190301", "400"), ()) + * ) + */ + @Test (expected = AnalysisException.class) + public void testFixedRange3() throws DdlException, AnalysisException { + //add columns + int columns = 2; + Column k1 = new Column("k1", new ScalarType(PrimitiveType.INT), true, null, "", ""); + Column k2 = new Column("k2", new ScalarType(PrimitiveType.BIGINT), true, null, "", ""); + partitionColumns.add(k1); + partitionColumns.add(k2); + + //add RangePartitionDescs + PartitionKeyDesc p1 = new PartitionKeyDesc( + Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("200")), + new ArrayList<>()); + + singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", p1, null)); + + partitionInfo = new RangePartitionInfo(partitionColumns); + + for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) { + singleRangePartitionDesc.analyze(columns, null); + partitionInfo.handleNewSinglePartitionDesc(singleRangePartitionDesc, 20000L); + } + } + + /** + * PARTITION BY RANGE(`k1`, `k2`) ( + * PARTITION p0 VALUES [("20190101", "100"),("20190201")) + * ) + */ + @Test + public void testFixedRange4() throws DdlException, AnalysisException { + //add columns + int columns = 2; + Column k1 = new Column("k1", new ScalarType(PrimitiveType.INT), true, null, "", ""); + Column k2 = new Column("k2", new ScalarType(PrimitiveType.BIGINT), true, null, "", ""); + partitionColumns.add(k1); + partitionColumns.add(k2); + + //add RangePartitionDescs + PartitionKeyDesc p1 = new PartitionKeyDesc( + Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("100")), + Lists.newArrayList(new PartitionValue("20190201"))); + + singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", p1, null)); + + partitionInfo = new RangePartitionInfo(partitionColumns); + + for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) { + singleRangePartitionDesc.analyze(columns, null); + partitionInfo.handleNewSinglePartitionDesc(singleRangePartitionDesc, 20000L); + } + } + + /** + * 失败用例 + * PARTITION BY RANGE(`k1`, `k2`) ( + * PARTITION p0 VALUES [("20190101", "100"),("20190101", "100")) + * ) + */ + @Test (expected = DdlException.class) + public void testFixedRange5() throws DdlException, AnalysisException { + //add columns + int columns = 2; + Column k1 = new Column("k1", new ScalarType(PrimitiveType.INT), true, null, "", ""); + Column k2 = new Column("k2", new ScalarType(PrimitiveType.BIGINT), true, null, "", ""); + partitionColumns.add(k1); + partitionColumns.add(k2); + + //add RangePartitionDescs + PartitionKeyDesc p1 = new PartitionKeyDesc( + Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("100")), + Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("100"))); + + singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", p1, null)); + + partitionInfo = new RangePartitionInfo(partitionColumns); + + for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) { + singleRangePartitionDesc.analyze(columns, null); + partitionInfo.handleNewSinglePartitionDesc(singleRangePartitionDesc, 20000L); + } + } + } diff --git a/fe/src/test/java/org/apache/doris/es/EsStateStoreTest.java b/fe/src/test/java/org/apache/doris/es/EsStateStoreTest.java index b6cfbac8fb..030dfba9bc 100644 --- a/fe/src/test/java/org/apache/doris/es/EsStateStoreTest.java +++ b/fe/src/test/java/org/apache/doris/es/EsStateStoreTest.java @@ -22,6 +22,7 @@ import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; +import org.apache.doris.analysis.PartitionValue; import org.apache.doris.catalog.Catalog; import org.apache.doris.catalog.CatalogTestUtil; import org.apache.doris.catalog.EsTable; @@ -41,7 +42,6 @@ import com.google.common.collect.Range; import org.junit.Before; import org.junit.BeforeClass; -import org.junit.Ignore; import org.junit.Test; import java.io.BufferedReader; @@ -119,16 +119,16 @@ public class EsStateStoreTest { assertEquals(5, esIndexState1.getShardRoutings().size()); assertEquals("index1", esIndexState1.getIndexName()); PartitionKey lowKey = PartitionKey.createInfinityPartitionKey(definedPartInfo.getPartitionColumns(), false); - PartitionKey upperKey = PartitionKey.createPartitionKey(Lists.newArrayList("2018-10-01"), + PartitionKey upperKey = PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("2018-10-01")), definedPartInfo.getPartitionColumns()); Range newRange = Range.closedOpen(lowKey, upperKey); assertEquals(newRange, part0); Range part1 = rangeMap.get(new Long(1)); EsIndexState esIndexState2 = esTableState.getIndexState(1); assertEquals("index2", esIndexState2.getIndexName()); - lowKey = PartitionKey.createPartitionKey(Lists.newArrayList("2018-10-01"), + lowKey = PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("2018-10-01")), definedPartInfo.getPartitionColumns()); - upperKey = PartitionKey.createPartitionKey(Lists.newArrayList("2018-10-02"), + upperKey = PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("2018-10-02")), definedPartInfo.getPartitionColumns()); newRange = Range.closedOpen(lowKey, upperKey); assertEquals(newRange, part1); @@ -171,7 +171,7 @@ public class EsStateStoreTest { assertEquals(5, esIndexState1.getShardRoutings().size()); assertEquals("index1", esIndexState1.getIndexName()); PartitionKey lowKey = PartitionKey.createInfinityPartitionKey(definedPartInfo.getPartitionColumns(), false); - PartitionKey upperKey = PartitionKey.createPartitionKey(Lists.newArrayList("2018-10-01"), + PartitionKey upperKey = PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("2018-10-01")), definedPartInfo.getPartitionColumns()); Range newRange = Range.closedOpen(lowKey, upperKey); assertEquals(newRange, part0); diff --git a/fe/src/test/java/org/apache/doris/planner/OlapTableSinkTest.java b/fe/src/test/java/org/apache/doris/planner/OlapTableSinkTest.java index 9aad7920cf..8d422bf015 100644 --- a/fe/src/test/java/org/apache/doris/planner/OlapTableSinkTest.java +++ b/fe/src/test/java/org/apache/doris/planner/OlapTableSinkTest.java @@ -18,6 +18,7 @@ package org.apache.doris.planner; import org.apache.doris.analysis.DescriptorTable; +import org.apache.doris.analysis.PartitionValue; import org.apache.doris.analysis.SlotDescriptor; import org.apache.doris.analysis.TupleDescriptor; import org.apache.doris.catalog.Column; @@ -115,7 +116,7 @@ public class OlapTableSinkTest { 2, Lists.newArrayList(new Column("k1", PrimitiveType.BIGINT))); Column partKey = new Column("k2", PrimitiveType.VARCHAR); - PartitionKey key = PartitionKey.createPartitionKey(Lists.newArrayList("123"), Lists.newArrayList(partKey)); + PartitionKey key = PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("123")), Lists.newArrayList(partKey)); Partition p1 = new Partition(1, "p1", index, distInfo); Partition p2 = new Partition(2, "p2", index, distInfo); diff --git a/fe/src/test/java/org/apache/doris/task/AgentTaskTest.java b/fe/src/test/java/org/apache/doris/task/AgentTaskTest.java index 317dd8650b..b761e6f812 100644 --- a/fe/src/test/java/org/apache/doris/task/AgentTaskTest.java +++ b/fe/src/test/java/org/apache/doris/task/AgentTaskTest.java @@ -17,6 +17,7 @@ package org.apache.doris.task; +import org.apache.doris.analysis.PartitionValue; import org.apache.doris.catalog.AggregateType; import org.apache.doris.catalog.Column; import org.apache.doris.catalog.KeysType; @@ -98,7 +99,7 @@ public class AgentTaskTest { columns.add(new Column("v1", ScalarType.createType(PrimitiveType.INT), false, AggregateType.SUM, "1", "")); PartitionKey pk1 = PartitionKey.createInfinityPartitionKey(Arrays.asList(columns.get(0)), false); - PartitionKey pk2 = PartitionKey.createPartitionKey(Arrays.asList("10"), Arrays.asList(columns.get(0))); + PartitionKey pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("10")), Arrays.asList(columns.get(0))); range1 = Range.closedOpen(pk1, pk2); PartitionKey pk3 = PartitionKey.createInfinityPartitionKey(Arrays.asList(columns.get(0)), true);