database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

Support adding fixed range partition

Browse Source 
eg: ALTER TABLE test_table ADD PARTITION p0125 VALUES [("20190125"), ("20190126"));
This commit is contained in:
worker24h
2019-10-14 20:50:30 -05:00
committed by Mingyu Chen
parent 62acf5d098
commit ec7c8a2c6f
22 changed files with 612 additions and 160 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/documentation/cn/administrator-guide/load-data/routine-load-manual.md
View File

@ -63,7 +63,7 @@ FE 中的 JobScheduler 根据汇报结果，继续生成后续新的 Task，或
### 创建例行导入任务
创建例行导入任务的的详细语法可以连接到 Doris 后，执行 `HELP CREATE ROUTINE LOAD;` 查看语法帮助。这里主要详细介绍，创建作业时的注意事项。
创建例行导入任务的的详细语法可以连接到 Doris 后，执行 `HELP ROUTINE LOAD;` 查看语法帮助。这里主要详细介绍，创建作业时的注意事项。
* columns_mapping

41
docs/documentation/cn/sql-reference/sql-statements/Data Definition/CREATE TABLE.md
View File

@ -112,12 +112,13 @@
除AGGREGATE KEY外，其他key_type在建表时，value列不需要指定聚合类型。
4. partition_desc
1) Range 分区
partition描述有两种使用方式
1) LESS THAN
语法：
PARTITION BY RANGE (k1, k2, ...)
(
PARTITION partition_name VALUES LESS THAN MAXVALUE|("value1", "value2", ...)
PARTITION partition_name VALUES LESS THAN MAXVALUE|("value1", "value2", ...)
PARTITION partition_name1 VALUES LESS THAN MAXVALUE|("value1", "value2", ...),
PARTITION partition_name2 VALUES LESS THAN MAXVALUE|("value1", "value2", ...)
...
)
说明：
@ -132,6 +133,17 @@
注意：
1) 分区一般用于时间维度的数据管理
2) 有数据回溯需求的，可以考虑首个分区为空分区，以便后续增加分区
2）Fixed Range
语法：
PARTITION BY RANGE (k1, k2, k3, ...)
(
PARTITION partition_name1 VALUES [("k1-lower1", "k2-lower1", "k3-lower1",...), ("k1-upper1", "k2-upper1", "k3-upper1", ...)),
PARTITION partition_name2 VALUES [("k1-lower1-2", "k2-lower1-2", ...), ("k1-upper1-2", MAXVALUE, ))
"k3-upper1-2", ...
)
说明：
1）Fixed Range比LESS THAN相对灵活些，左右区间完全由用户自己确定
2）其他与LESS THAN保持同步
5. distribution_desc
1) Hash 分桶
@ -216,6 +228,7 @@
3. 创建一个 olap 表，使用 Key Range 分区，使用Hash分桶，默认使用列存，
相同key的记录同时存在，设置初始存储介质和冷却时间
1）LESS THAN
CREATE TABLE example_db.table_range
(
k1 DATE,
@ -244,7 +257,27 @@
[ {"2014-06-01"}, {"2014-12-01"} )
不在这些分区范围内的数据将视为非法数据被过滤
2) Fixed Range
CREATE TABLE table_range
(
k1 DATE,
k2 INT,
k3 SMALLINT,
v1 VARCHAR(2048),
v2 DATETIME DEFAULT "2014-02-04 15:36:00"
)
ENGINE=olap
DUPLICATE KEY(k1, k2, k3)
PARTITION BY RANGE (k1, k2, k3)
(
PARTITION p1 VALUES [("2014-01-01", "10", "200"), ("2014-01-01", "20", "300")),
PARTITION p2 VALUES [("2014-06-01", "100", "200"), ("2014-07-01", "100", "300"))
)
DISTRIBUTED BY HASH(k2) BUCKETS 32
PROPERTIES(
"storage_medium" = "SSD"
);
4. 创建一个 mysql 表
CREATE TABLE example_db.table_mysql
(

2
docs/documentation/en/administrator-guide/load-data/routine-load-manual_EN.md
View File

@ -63,7 +63,7 @@ Currently we only support routine imports from Kafka systems. This section detai
### Create routine import tasks
After creating a detailed grammar for routine import tasks, you can connect to Doris and execute `HELP CREATE ROUTINE LOAD'; `Check grammar help. Here is a detailed description of the creation of the job note.
After creating a detailed grammar for routine import tasks, you can connect to Doris and execute `HELP ROUTINE LOAD'; `Check grammar help. Here is a detailed description of the creation of the job note.
* columns_mapping

49
docs/documentation/en/sql-reference/sql-statements/Data Definition/CREATE TABLE_EN.md
View File

@ -139,16 +139,16 @@
Except for AGGREGATE KEY, no need to specify aggregation type for value columns.
4. partition_desc
1) Range partition
Partition has two ways to use:
1) LESS THAN
Syntex:
```
PARTITION BY RANGE (k1, k2, ...)
(
PARTITION partition_name VALUES LESS THAN MAXVALUE|("value1", "value2", ...)
PARTITION partition_name VALUES LESS THAN MAXVALUE|("value1", "value2", ...)
PARTITION partition_name1 VALUES LESS THAN MAXVALUE|("value1", "value2", ...),
PARTITION partition_name2 VALUES LESS THAN MAXVALUE|("value1", "value2", ...)
...
)
```
@ -161,7 +161,23 @@
3) The range is [closed, open). And the lower bound of first partition is MIN VALUE of specifed column type.
4) NULL values should be save in partition which includes MIN VALUE.
5) Support multi partition columns, the the default partition value is MIN VALUE.
2）Fixed Range
Syntex:
```
PARTITION BY RANGE (k1, k2, k3, ...)
(
PARTITION partition_name1 VALUES [("k1-lower1", "k2-lower1", "k3-lower1",...), ("k1-upper1", "k2-upper1", "k3-upper1", ...)),
PARTITION partition_name2 VALUES [("k1-lower1-2", "k2-lower1-2", ...), ("k1-upper1-2", MAXVALUE, ))
"k3-upper1-2", ...
)
```
Explain:
1）The Fixed Range is more flexible than the LESS THAN, and the left and right intervals are completely determined by the user.
2）Others are consistent with LESS THAN.
5. distribution_desc
1) Hash
Syntax:
@ -248,7 +264,7 @@
);
3. Create an olap table, with range partitioned, distributed by hash.
1) LESS THAN
```
CREATE TABLE example_db.table_range
(
@ -283,6 +299,27 @@
Data outside these ranges will not be loaded.
2) Fixed Range
CREATE TABLE table_range
(
k1 DATE,
k2 INT,
k3 SMALLINT,
v1 VARCHAR(2048),
v2 DATETIME DEFAULT "2014-02-04 15:36:00"
)
ENGINE=olap
DUPLICATE KEY(k1, k2, k3)
PARTITION BY RANGE (k1, k2, k3)
(
PARTITION p1 VALUES [("2014-01-01", "10", "200"), ("2014-01-01", "20", "300")),
PARTITION p2 VALUES [("2014-06-01", "100", "200"), ("2014-07-01", "100", "300"))
)
DISTRIBUTED BY HASH(k2) BUCKETS 32
PROPERTIES(
"storage_medium" = "SSD"
);
4. Create a mysql table
```

43
fe/src/main/cup/sql_parser.cup
View File

@ -366,6 +366,8 @@ nonterminal Long opt_field_length;
nonterminal KeysDesc opt_keys;
nonterminal PartitionKeyDesc partition_key_desc;
nonterminal PartitionKeyDesc fixed_partition_key_desc;
nonterminal List<PartitionValue> partition_key_list;
nonterminal SingleRangePartitionDesc single_range_partition_desc;
nonterminal List<SingleRangePartitionDesc> opt_single_range_partition_desc_list;
nonterminal List<SingleRangePartitionDesc> single_range_partition_desc_list;
@ -1545,6 +1547,11 @@ single_range_partition_desc ::=
{:
RESULT = new SingleRangePartitionDesc(ifNotExists, partName, desc, properties);
:}
| KW_PARTITION opt_if_not_exists:ifNotExists ident:partName KW_VALUES fixed_partition_key_desc:desc
opt_key_value_map:properties
{:
RESULT = new SingleRangePartitionDesc(ifNotExists, partName, desc, properties);
:}
;
partition_key_desc ::=
@ -1552,12 +1559,46 @@ partition_key_desc ::=
{:
RESULT = PartitionKeyDesc.createMaxKeyDesc();
:}
| LPAREN string_list:keys RPAREN
| LPAREN partition_key_list:keys RPAREN
{:
RESULT = new PartitionKeyDesc(keys);
:}
;
partition_key_list ::=
/* empty */
{:
List<PartitionValue> l = new ArrayList<PartitionValue>();
RESULT = l;
:}
| partition_key_list:l COMMA STRING_LITERAL:item
{:
l.add(new PartitionValue(item));
RESULT = l;
:}
| partition_key_list:l COMMA KW_MAX_VALUE
{:
l.add(PartitionValue.createMaxValue());
RESULT = l;
:}
| STRING_LITERAL:item
{:
RESULT = Lists.newArrayList(new PartitionValue(item));
:}
| KW_MAX_VALUE
{:
RESULT = Lists.newArrayList(PartitionValue.createMaxValue());
:}
;
fixed_partition_key_desc ::=
/* format: [(lower), (upper))*/
LBRACKET LPAREN partition_key_list:lower RPAREN COMMA LPAREN partition_key_list:upper RPAREN RPAREN
{:
RESULT = new PartitionKeyDesc(lower, upper);
:}
;
opt_engine ::=
{: RESULT = null; :}
| KW_ENGINE EQUAL ident:engineName

94
fe/src/main/java/org/apache/doris/analysis/PartitionKeyDesc.java
View File

@ -32,86 +32,122 @@ import java.util.List;
// 用于表达在创建表、创建rollup中key range partition中所使用的key信息
// 在知道具体列信息后，可以转成PartitionKey，仅仅在语法解析中短暂的有意义
public class PartitionKeyDesc implements Writable {
// public static final PartitionKeyDesc MAX_VALUE = new PartitionKeyDesc();
// lower values only be used for restore
private List<String> lowerValues;
private List<String> upperValues;
public enum PartitionRangeType {
INVALID,
LESS_THAN,
FIXED
}
private List<PartitionValue> lowerValues;
private List<PartitionValue> upperValues;
private PartitionRangeType partitionType;
public static PartitionKeyDesc createMaxKeyDesc() {
return new PartitionKeyDesc();
}
public PartitionKeyDesc() {
lowerValues = Lists.newArrayList();
upperValues = Lists.newArrayList();
partitionType = PartitionRangeType.LESS_THAN; //LESS_THAN is default type.
}
// used by SQL parser
public PartitionKeyDesc(List<String> upperValues) {
this.lowerValues = Lists.newArrayList();
public PartitionKeyDesc(List<PartitionValue> upperValues) {
this.upperValues = upperValues;
partitionType = PartitionRangeType.LESS_THAN;
}
public PartitionKeyDesc(List<String> lowerValues, List<String> upperValues) {
public PartitionKeyDesc(List<PartitionValue> lowerValues, List<PartitionValue> upperValues) {
this.lowerValues = lowerValues;
this.upperValues = upperValues;
partitionType = PartitionRangeType.FIXED;
}
public void setLowerValues(List<String> lowerValues) {
public void setLowerValues(List<PartitionValue> lowerValues) {
this.lowerValues = lowerValues;
}
public List<String> getLowerValues() {
public List<PartitionValue> getLowerValues() {
return lowerValues;
}
public List<String> getUpperValues() {
public List<PartitionValue> getUpperValues() {
return upperValues;
}
public boolean isMax() {
return upperValues.isEmpty();
return lowerValues == null && upperValues == null;
}
public boolean hasLowerValues() {
return lowerValues != null;
}
public boolean hasUpperValues() {
return upperValues != null;
}
public PartitionRangeType getPartitionType () {
return partitionType;
}
public String toSql() {
if (this.isMax()) {
return "MAXVALUE";
}
StringBuilder sb = new StringBuilder("(");
Joiner.on(", ").appendTo(sb, Lists.transform(upperValues, new Function<String, String>() {
@Override
public String apply(String s) {
return "'" + s + "'";
}
})).append(")");
return sb.toString();
if (upperValues != null) {
StringBuilder sb = new StringBuilder("(");
Joiner.on(", ").appendTo(sb, Lists.transform(upperValues, new Function<PartitionValue, String>() {
@Override
public String apply(PartitionValue v) {
return "'" + v.getStringValue() + "'";
}
})).append(")");
return sb.toString();
} else {
return "()";
}
}
@Override
public void write(DataOutput out) throws IOException {
int count = lowerValues.size();
int count = lowerValues == null ? 0 : lowerValues.size();
out.writeInt(count);
for (String value : lowerValues) {
Text.writeString(out, value);
if (count > 0) {
for (PartitionValue value : lowerValues) {
Text.writeString(out, value.getStringValue());
}
}
count = upperValues.size();
count = upperValues == null ? 0: upperValues.size();
out.writeInt(count);
for (String value : upperValues) {
Text.writeString(out, value);
if (count > 0) {
for (PartitionValue value : upperValues) {
Text.writeString(out, value.getStringValue());
}
}
}
@Override
public void readFields(DataInput in) throws IOException {
int count = in.readInt();
for (int i = 0; i < count; i++) {
lowerValues.add(Text.readString(in));
String v = Text.readString(in);
if (v.equals("MAXVALUE")) {
lowerValues.add(new PartitionValue());
} else {
lowerValues.add(new PartitionValue(v));
}
}
count = in.readInt();
for (int i = 0; i < count; i++) {
upperValues.add(Text.readString(in));
String v = Text.readString(in);
if (v.equals("MAXVALUE")) {
upperValues.add(new PartitionValue());
} else {
upperValues.add(new PartitionValue(v));
}
}
}
}

62
fe/src/main/java/org/apache/doris/analysis/PartitionValue.java Normal file
View File

@ -0,0 +1,62 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
package org.apache.doris.analysis;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
public class PartitionValue {
private String value;
private boolean isMaxValue;
public static PartitionValue createMaxValue() {
PartitionValue value = new PartitionValue();
value.isMaxValue = true;
value.value = null;
return value;
}
public PartitionValue() {
}
public PartitionValue(String value) {
this.isMaxValue = false;
this.value = value;
}
public LiteralExpr getValue(Type type) throws AnalysisException {
if (isMaxValue) {
return LiteralExpr.createInfinity(type, true);
} else {
return LiteralExpr.create(value, type);
}
}
public boolean isMax() {
return isMaxValue;
}
public String getStringValue() {
if (isMaxValue) {
return "MAXVALUE";
} else {
return value;
}
}
}

8
fe/src/main/java/org/apache/doris/analysis/RangePartitionDesc.java
View File

@ -21,6 +21,7 @@ import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.PartitionInfo;
import org.apache.doris.catalog.PartitionType;
import org.apache.doris.catalog.RangePartitionInfo;
import org.apache.doris.analysis.PartitionKeyDesc.PartitionRangeType;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.io.Text;
@ -94,6 +95,7 @@ public class RangePartitionDesc extends PartitionDesc {
}
Set<String> nameSet = Sets.newTreeSet(String.CASE_INSENSITIVE_ORDER);
PartitionRangeType partitionType = PartitionRangeType.INVALID;
for (SingleRangePartitionDesc desc : singleRangePartitionDescs) {
if (!nameSet.add(desc.getPartitionName())) {
throw new AnalysisException("Duplicated partition name: " + desc.getPartitionName());
@ -104,6 +106,12 @@ public class RangePartitionDesc extends PartitionDesc {
if (otherProperties != null) {
givenProperties = Maps.newHashMap(otherProperties);
}
// check partitionType
if (partitionType == PartitionRangeType.INVALID) {
partitionType = desc.getPartitionKeyDesc().getPartitionType();
} else if (partitionType != desc.getPartitionKeyDesc().getPartitionType()) {
throw new AnalysisException("You can only use one of these methods to create partitions");
}
desc.analyze(columnDefs.size(), givenProperties);
}
}

50
fe/src/main/java/org/apache/doris/catalog/PartitionKey.java
View File

@ -22,6 +22,7 @@ import org.apache.doris.analysis.IntLiteral;
import org.apache.doris.analysis.LargeIntLiteral;
import org.apache.doris.analysis.LiteralExpr;
import org.apache.doris.analysis.MaxLiteral;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
@ -62,13 +63,13 @@ public class PartitionKey implements Comparable<PartitionKey>, Writable {
return partitionKey;
}
public static PartitionKey createPartitionKey(List<String> keys, List<Column> columns)
public static PartitionKey createPartitionKey(List<PartitionValue> keys, List<Column> columns)
throws AnalysisException {
PartitionKey partitionKey = new PartitionKey();
Preconditions.checkArgument(keys.size() <= columns.size());
int i;
for (i = 0; i < keys.size(); ++i) {
partitionKey.keys.add(LiteralExpr.create(keys.get(i),
partitionKey.keys.add(keys.get(i).getValue(
Type.fromPrimitiveType(columns.get(i).getDataType())));
partitionKey.types.add(columns.get(i).getDataType());
}
@ -127,6 +128,29 @@ public class PartitionKey implements Comparable<PartitionKey>, Writable {
return true;
}
public static int compareLiteralExpr(LiteralExpr key1, LiteralExpr key2) {
int ret = 0;
if (key1 instanceof MaxLiteral || key2 instanceof MaxLiteral) {
ret = key1.compareLiteral(key2);
} else {
final Type destType = Type.getAssignmentCompatibleType(key1.getType(), key2.getType(), false);
try {
LiteralExpr newKey = key1;
if (key1.getType() != destType) {
newKey = (LiteralExpr) key1.castTo(destType);
}
LiteralExpr newOtherKey = key2;
if (key2.getType() != destType) {
newOtherKey = (LiteralExpr) key2.castTo(destType);
}
ret = newKey.compareLiteral(newOtherKey);
} catch (AnalysisException e) {
throw new RuntimeException("Cast error in partition");
}
}
return ret;
}
// compare with other PartitionKey. used for partition prune
@Override
public int compareTo(PartitionKey other) {
@ -134,27 +158,7 @@ public class PartitionKey implements Comparable<PartitionKey>, Writable {
int other_key_len = other.keys.size();
int min_len = Math.min(this_key_len, other_key_len);
for (int i = 0; i < min_len; ++i) {
final LiteralExpr oldKey = this.getKeys().get(i);
final LiteralExpr otherOldKey = other.getKeys().get(i);
int ret = 0;
if (oldKey instanceof MaxLiteral || otherOldKey instanceof MaxLiteral) {
ret = oldKey.compareLiteral(otherOldKey);
} else {
final Type destType = Type.getAssignmentCompatibleType(oldKey.getType(), otherOldKey.getType(), false);
try {
LiteralExpr newKey = oldKey;
if (oldKey.getType() != destType) {
newKey = (LiteralExpr) oldKey.castTo(destType);
}
LiteralExpr newOtherKey = otherOldKey;
if (otherOldKey.getType() != destType) {
newOtherKey = (LiteralExpr) otherOldKey.castTo(destType);
}
ret = newKey.compareLiteral(newOtherKey);
} catch (AnalysisException e) {
throw new RuntimeException("Cast error in partition");
}
}
int ret = compareLiteralExpr(this.getKeys().get(i), other.getKeys().get(i));
if (0 != ret) {
return ret;
}

30
fe/src/main/java/org/apache/doris/catalog/RangePartitionInfo.java
View File

@ -18,6 +18,7 @@
package org.apache.doris.catalog;
import org.apache.doris.analysis.PartitionKeyDesc;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.analysis.SingleRangePartitionDesc;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.DdlException;
@ -122,7 +123,7 @@ public class RangePartitionInfo extends PartitionInfo {
PartitionKey upperKey = nextRange.upperEndpoint();
if (upperKey.compareTo(singlePartitionKey) >= 0) {
PartitionKey lowKey = null;
if (!partKeyDesc.getLowerValues().isEmpty()) {
if (partKeyDesc.hasLowerValues()) {
lowKey = PartitionKey.createPartitionKey(partKeyDesc.getLowerValues(), partitionColumns);
} else {
if (lastRange == null) {
@ -131,7 +132,10 @@ public class RangePartitionInfo extends PartitionInfo {
lowKey = lastRange.upperEndpoint();
}
}
// check: [left, right), error if left equal right
if (lowKey.compareTo(singlePartitionKey) >= 0) {
throw new IllegalArgumentException("The right value must be more than the left value");
}
newRange = Range.closedOpen(lowKey, singlePartitionKey);
// check if range intersected
@ -143,7 +147,7 @@ public class RangePartitionInfo extends PartitionInfo {
if (newRange == null) {
PartitionKey lowKey = null;
if (!partKeyDesc.getLowerValues().isEmpty()) {
if (partKeyDesc.hasLowerValues()) {
lowKey = PartitionKey.createPartitionKey(partKeyDesc.getLowerValues(), partitionColumns);
} else {
if (lastRange == null) {
@ -153,8 +157,16 @@ public class RangePartitionInfo extends PartitionInfo {
lowKey = lastRange.upperEndpoint();
}
}
// check: [left, right), error if left equal right
if (lowKey.compareTo(singlePartitionKey) >= 0) {
throw new IllegalArgumentException("The right value must be more than the left value");
}
newRange = Range.closedOpen(lowKey, singlePartitionKey);
// check if range intersected. The first Partition if lastRange == null
if (lastRange != null) {
checkRangeIntersect(newRange, lastRange);
}
}
} catch (AnalysisException e) {
throw new DdlException("Invalid range value format： " + e.getMessage());
@ -181,7 +193,7 @@ public class RangePartitionInfo extends PartitionInfo {
idToRange.put(partitionId, range);
} catch (IllegalArgumentException e) {
// Range.closedOpen may throw this if (lower > upper)
throw new DdlException("Invalid key range", e);
throw new DdlException("Invalid key range: " + e.getMessage());
}
idToDataProperty.put(partitionId, desc.getPartitionDataProperty());
idToReplicationNum.put(partitionId, desc.getReplicationNum());
@ -298,10 +310,10 @@ public class RangePartitionInfo extends PartitionInfo {
public SingleRangePartitionDesc toSingleRangePartitionDesc(long partitionId, String partitionName,
Map<String, String> properties) {
Range<PartitionKey> range = idToRange.get(partitionId);
List<String> upperValues = Lists.newArrayList();
List<String> lowerValues = Lists.newArrayList();
List<PartitionValue> upperValues = Lists.newArrayList();
List<PartitionValue> lowerValues = Lists.newArrayList();
// FIXME(cmy): check here(getStringValue)
lowerValues.add(range.lowerEndpoint().getKeys().get(0).getStringValue());
lowerValues.add(new PartitionValue(range.lowerEndpoint().getKeys().get(0).getStringValue()));
PartitionKey upperKey = range.upperEndpoint();
PartitionKeyDesc keyDesc = null;
@ -309,7 +321,7 @@ public class RangePartitionInfo extends PartitionInfo {
keyDesc = PartitionKeyDesc.createMaxKeyDesc();
keyDesc.setLowerValues(lowerValues);
} else {
upperValues.add(range.upperEndpoint().getKeys().get(0).getStringValue());
upperValues.add(new PartitionValue(range.upperEndpoint().getKeys().get(0).getStringValue()));
keyDesc = new PartitionKeyDesc(lowerValues, upperValues);
}

20
fe/src/main/java/org/apache/doris/common/util/KuduUtil.java
View File

@ -26,8 +26,8 @@ import org.apache.doris.analysis.RangePartitionDesc;
import org.apache.doris.analysis.SingleRangePartitionDesc;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.catalog.KuduPartition.KuduRange;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
@ -35,22 +35,22 @@ import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.FeNameFormat;
import java.nio.charset.Charset;
import java.util.List;
import java.util.Set;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
import org.apache.kudu.ColumnSchema;
import org.apache.kudu.Type;
import org.apache.kudu.Schema;
import org.apache.kudu.Type;
import org.apache.kudu.client.CreateTableOptions;
import org.apache.kudu.client.KuduClient;
import org.apache.kudu.client.KuduClient.KuduClientBuilder;
import org.apache.kudu.client.PartialRow;
import org.apache.kudu.client.RangePartitionBound;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
import java.nio.charset.Charset;
import java.util.List;
import java.util.Set;
public class KuduUtil {
@ -105,8 +105,8 @@ public class KuduUtil {
PartialRow lower = kuduSchema.newPartialRow();
PartialRow upper = kuduSchema.newPartialRow();
String lowerValue = lastDesc != null ? lastDesc.getPartitionKeyDesc().getUpperValues().get(0) : "";
String upperValue = single.getPartitionKeyDesc().getUpperValues().get(0);
String lowerValue = lastDesc != null ? lastDesc.getPartitionKeyDesc().getUpperValues().get(0).getStringValue() : "";
String upperValue = single.getPartitionKeyDesc().getUpperValues().get(0).getStringValue();
switch (kuduType) {
case INT8:

24
fe/src/main/java/org/apache/doris/external/EsIndexState.java vendored
View File

@ -17,27 +17,27 @@
package org.apache.doris.external;
import java.util.List;
import java.util.Map;
import java.util.Random;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.json.JSONArray;
import org.json.JSONObject;
import org.apache.doris.analysis.PartitionKeyDesc;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.analysis.SingleRangePartitionDesc;
import org.apache.doris.catalog.PartitionInfo;
import org.apache.doris.catalog.PartitionKey;
import org.apache.doris.catalog.RangePartitionInfo;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.thrift.TNetworkAddress;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.json.JSONArray;
import org.json.JSONObject;
import java.util.List;
import java.util.Map;
import java.util.Random;
public class EsIndexState {
private static final Logger LOG = LogManager.getLogger(EsIndexState.class);
@ -112,7 +112,7 @@ public class EsIndexState {
partitionSetting == null ? "" : partitionSetting.toString());
if (partitionSetting != null && partitionSetting.has("upperbound")) {
String upperBound = partitionSetting.getString("upperbound");
List<String> upperValues = Stream.of(upperBound).collect(Collectors.toList());
List<PartitionValue> upperValues = Lists.newArrayList(new PartitionValue(upperBound));
PartitionKeyDesc partitionKeyDesc = new PartitionKeyDesc(upperValues);
// use index name as partition name
SingleRangePartitionDesc desc = new SingleRangePartitionDesc(false,

7
fe/src/main/java/org/apache/doris/planner/PartitionColumnFilter.java
View File

@ -19,6 +19,7 @@ package org.apache.doris.planner;
import org.apache.doris.analysis.InPredicate;
import org.apache.doris.analysis.LiteralExpr;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.PartitionKey;
import org.apache.doris.common.AnalysisException;
@ -98,8 +99,10 @@ public class PartitionColumnFilter {
PartitionKey upperKey = null;
// cmy mod, catch AnalysisException
try {
lowerKey = PartitionKey.createPartitionKey(Lists.newArrayList(lowerBound.getStringValue()), columns);
upperKey = PartitionKey.createPartitionKey(Lists.newArrayList(upperBound.getStringValue()), columns);
lowerKey = PartitionKey.createPartitionKey(
Lists.newArrayList(new PartitionValue(lowerBound.getStringValue())), columns);
upperKey = PartitionKey.createPartitionKey(
Lists.newArrayList(new PartitionValue(upperBound.getStringValue())), columns);
} catch (AnalysisException e) {
LOG.warn(e.getMessage());
return null;

8
fe/src/test/java/org/apache/doris/analysis/PartitionKeyDescTest.java
View File

@ -25,12 +25,12 @@ import org.junit.Test;
import java.util.List;
public class PartitionKeyDescTest {
private List<String> values;
private List<PartitionValue> values;
// value of key is ["1", "abc"]
@Before
public void setUp() {
values = Lists.newArrayList("1", "abc");
values = Lists.newArrayList(new PartitionValue("1"), new PartitionValue("abc"));
}
@Test
@ -45,7 +45,7 @@ public class PartitionKeyDescTest {
public void testMax() {
PartitionKeyDesc desc = PartitionKeyDesc.createMaxKeyDesc();
Assert.assertTrue(desc.getUpperValues().isEmpty());
Assert.assertNull(desc.getUpperValues());
Assert.assertEquals("MAXVALUE", desc.toSql());
}
}
}

3
fe/src/test/java/org/apache/doris/backup/AlterClauseRWTest.java
View File

@ -23,6 +23,7 @@ import org.apache.doris.analysis.AlterClause;
import org.apache.doris.analysis.AlterTableStmt;
import org.apache.doris.analysis.DistributionDesc;
import org.apache.doris.analysis.PartitionKeyDesc;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.analysis.RandomDistributionDesc;
import org.apache.doris.analysis.SingleRangePartitionDesc;
import org.apache.doris.analysis.TableName;
@ -64,7 +65,7 @@ public class AlterClauseRWTest {
// add partition clause
String partititionName = "p1";
List<String> values = Lists.newArrayList("100");
List<PartitionValue> values = Lists.newArrayList(new PartitionValue("100"));
PartitionKeyDesc keyDesc = new PartitionKeyDesc(values);
Map<String, String> properties = Maps.newHashMap();
SingleRangePartitionDesc partitionDesc = new SingleRangePartitionDesc(false, partititionName, keyDesc,

7
fe/src/test/java/org/apache/doris/backup/CatalogMocker.java
View File

@ -19,6 +19,7 @@ package org.apache.doris.backup;
import org.apache.doris.alter.RollupHandler;
import org.apache.doris.alter.SchemaChangeHandler;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.catalog.AggregateType;
import org.apache.doris.catalog.Catalog;
import org.apache.doris.catalog.Column;
@ -282,16 +283,16 @@ public class CatalogMocker {
PartitionKey rangeP1Lower =
PartitionKey.createInfinityPartitionKey(Lists.newArrayList(TEST_TBL_BASE_SCHEMA.get(0)), false);
PartitionKey rangeP1Upper =
PartitionKey.createPartitionKey(Lists.newArrayList("10"),
PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("10")),
Lists.newArrayList(TEST_TBL_BASE_SCHEMA.get(0)));
Range<PartitionKey> rangeP1 = Range.closedOpen(rangeP1Lower, rangeP1Upper);
rangePartitionInfo.setRange(TEST_PARTITION1_ID, rangeP1);
PartitionKey rangeP2Lower =
PartitionKey.createPartitionKey(Lists.newArrayList("10"),
PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("10")),
Lists.newArrayList(TEST_TBL_BASE_SCHEMA.get(0)));
PartitionKey rangeP2Upper =
PartitionKey.createPartitionKey(Lists.newArrayList("20"),
PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("20")),
Lists.newArrayList(TEST_TBL_BASE_SCHEMA.get(0)));
Range<PartitionKey> rangeP2 = Range.closedOpen(rangeP2Lower, rangeP2Upper);
rangePartitionInfo.setRange(TEST_PARTITION2_ID, rangeP2);

5
fe/src/test/java/org/apache/doris/catalog/CatalogTestUtil.java
View File

@ -18,6 +18,7 @@
package org.apache.doris.catalog;
import org.apache.doris.analysis.PartitionKeyDesc;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.analysis.SingleRangePartitionDesc;
import org.apache.doris.catalog.MaterializedIndex.IndexExtState;
import org.apache.doris.catalog.MaterializedIndex.IndexState;
@ -248,7 +249,7 @@ public class CatalogTestUtil {
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1",
new PartitionKeyDesc(Lists
.newArrayList("100")),
.newArrayList(new PartitionValue("100"))),
null));
RangePartitionInfo partitionInfo = new RangePartitionInfo(partitionColumns);
@ -280,7 +281,7 @@ public class CatalogTestUtil {
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1",
new PartitionKeyDesc(Lists
.newArrayList("100")),
.newArrayList(new PartitionValue("100"))),
null));
SinglePartitionInfo partitionInfo = new SinglePartitionInfo();

64
fe/src/test/java/org/apache/doris/catalog/PartitionKeyTest.java
View File

@ -17,6 +17,7 @@
package org.apache.doris.catalog;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.common.AnalysisException;
import java.io.DataInputStream;
@ -78,77 +79,80 @@ public class PartitionKeyTest {
PartitionKey pk2;
// case1
pk1 = PartitionKey.createPartitionKey(Arrays.asList("127", "32767"),
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("32767")),
Arrays.asList(tinyInt, smallInt));
pk2 = PartitionKey.createInfinityPartitionKey(Arrays.asList(tinyInt, smallInt), true);
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case2
pk1 = PartitionKey.createPartitionKey(Arrays.asList("127"),
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127")),
Arrays.asList(tinyInt, smallInt));
pk2 = PartitionKey.createPartitionKey(Arrays.asList("127", "-32768"),
pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("-32768")),
Arrays.asList(tinyInt, smallInt));
Assert.assertTrue(pk1.equals(pk2) && pk1.compareTo(pk2) == 0);
// case3
pk1 = PartitionKey.createPartitionKey(Arrays.asList("127"),
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127")),
Arrays.asList(int32, bigInt));
pk2 = PartitionKey.createPartitionKey(Arrays.asList("128", "-32768"),
pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("128"), new PartitionValue("-32768")),
Arrays.asList(int32, bigInt));
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case4
pk1 = PartitionKey.createPartitionKey(Arrays.asList("127", "12345"),
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("12345")),
Arrays.asList(largeInt, bigInt));
pk2 = PartitionKey.createPartitionKey(Arrays.asList("127", "12346"),
pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("12346")),
Arrays.asList(largeInt, bigInt));
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case5
pk1 = PartitionKey.createPartitionKey(Arrays.asList("2014-12-12", "2014-12-12 10:00:00"),
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("2014-12-12"), new PartitionValue("2014-12-12 10:00:00")),
Arrays.asList(date, datetime));
pk2 = PartitionKey.createPartitionKey(Arrays.asList("2014-12-12", "2014-12-12 10:00:01"),
pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("2014-12-12"), new PartitionValue("2014-12-12 10:00:01")),
Arrays.asList(date, datetime));
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case6
pk1 = PartitionKey.createPartitionKey(Arrays.asList("-128"),
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("-128")),
Arrays.asList(tinyInt, smallInt));
pk2 = PartitionKey.createInfinityPartitionKey(Arrays.asList(tinyInt, smallInt), false);
Assert.assertTrue(pk1.equals(pk2) && pk1.compareTo(pk2) == 0);
// case7
pk1 = PartitionKey.createPartitionKey(Arrays.asList("127"),
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127")),
Arrays.asList(tinyInt, smallInt));
pk2 = PartitionKey.createInfinityPartitionKey(Arrays.asList(tinyInt, smallInt), true);
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case7
pk1 = PartitionKey.createPartitionKey(Arrays.asList("127", "32767"),
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("32767")),
Arrays.asList(tinyInt, smallInt));
pk2 = PartitionKey.createInfinityPartitionKey(Arrays.asList(tinyInt, smallInt), true);
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case8
pk1 = PartitionKey.createPartitionKey(Arrays.asList("127", "32767", "2147483647", "9223372036854775807",
"170141183460469231731687303715884105727",
"9999-12-31", "9999-12-31 23:59:59"),
allColumns);
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("32767"),
new PartitionValue("2147483647"), new PartitionValue("9223372036854775807"),
new PartitionValue("170141183460469231731687303715884105727"),
new PartitionValue("9999-12-31"), new PartitionValue("9999-12-31 23:59:59")),
allColumns);
pk2 = PartitionKey.createInfinityPartitionKey(allColumns, true);
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case9
pk1 = PartitionKey.createPartitionKey(Arrays.asList("-128", "-32768", "-2147483648", "-9223372036854775808",
"-170141183460469231731687303715884105728",
"1900-01-01", "1900-01-01 00:00:00"),
allColumns);
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("-128"), new PartitionValue("-32768"),
new PartitionValue("-2147483648"), new PartitionValue("-9223372036854775808"),
new PartitionValue("-170141183460469231731687303715884105728"),
new PartitionValue("1900-01-01"), new PartitionValue("1900-01-01 00:00:00")),
allColumns);
pk2 = PartitionKey.createInfinityPartitionKey(allColumns, false);
Assert.assertTrue(pk1.equals(pk2) && pk1.compareTo(pk2) == 0);
// case10
pk1 = PartitionKey.createPartitionKey(Arrays.asList("-128", "-32768", "0", "-9223372036854775808",
"0", "1970-01-01", "1970-01-01 00:00:00"),
allColumns);
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("-128"), new PartitionValue("-32768"),
new PartitionValue("0"), new PartitionValue("-9223372036854775808"),
new PartitionValue("0"), new PartitionValue("1970-01-01"), new PartitionValue("1970-01-01 00:00:00")),
allColumns);
pk2 = PartitionKey.createInfinityPartitionKey(allColumns, false);
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == 1);
}
@ -169,19 +173,19 @@ public class PartitionKeyTest {
PartitionKey keyEmpty = new PartitionKey();
keyEmpty.write(dos);
List<String> keys = new ArrayList<String>();
List<PartitionValue> keys = new ArrayList<PartitionValue>();
List<Column> columns = new ArrayList<Column>();
keys.add("100");
keys.add(new PartitionValue("100"));
columns.add(new Column("column2", ScalarType.createType(PrimitiveType.TINYINT), true, null, "", ""));
keys.add("101");
keys.add(new PartitionValue("101"));
columns.add(new Column("column3", ScalarType.createType(PrimitiveType.SMALLINT), true, null, "", ""));
keys.add("102");
keys.add(new PartitionValue("102"));
columns.add(new Column("column4", ScalarType.createType(PrimitiveType.INT), true, null, "", ""));
keys.add("103");
keys.add(new PartitionValue("103"));
columns.add(new Column("column5", ScalarType.createType(PrimitiveType.BIGINT), true, null, "", ""));
keys.add("2014-12-26");
keys.add(new PartitionValue("2014-12-26"));
columns.add(new Column("column10", ScalarType.createType(PrimitiveType.DATE), true, null, "", ""));
keys.add("2014-12-27 11:12:13");
keys.add(new PartitionValue("2014-12-27 11:12:13"));
columns.add(new Column("column11", ScalarType.createType(PrimitiveType.DATETIME), true, null, "", ""));
PartitionKey key = PartitionKey.createPartitionKey(keys, columns);

237
fe/src/test/java/org/apache/doris/catalog/RangePartitionInfoTest.java
View File

@ -18,6 +18,8 @@
package org.apache.doris.catalog;
import org.apache.doris.analysis.PartitionKeyDesc;
import org.apache.doris.analysis.PartitionKeyDesc.PartitionRangeType;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.analysis.SingleRangePartitionDesc;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.DdlException;
@ -27,6 +29,7 @@ import com.google.common.collect.Lists;
import org.junit.Before;
import org.junit.Test;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
@ -49,8 +52,8 @@ public class RangePartitionInfoTest {
partitionColumns.add(k1);
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1",
new PartitionKeyDesc(Lists .newArrayList("-128")),
null));
new PartitionKeyDesc(Lists .newArrayList(new PartitionValue("-128"))),
null));
partitionInfo = new RangePartitionInfo(partitionColumns);
@ -66,8 +69,8 @@ public class RangePartitionInfoTest {
partitionColumns.add(k1);
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1",
new PartitionKeyDesc(Lists.newArrayList("-32768")),
null));
new PartitionKeyDesc(Lists.newArrayList(new PartitionValue("-32768"))),
null));
partitionInfo = new RangePartitionInfo(partitionColumns);
for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) {
@ -82,9 +85,8 @@ public class RangePartitionInfoTest {
partitionColumns.add(k1);
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1",
new PartitionKeyDesc(Lists
.newArrayList("-2147483648")),
null));
new PartitionKeyDesc(Lists.newArrayList(new PartitionValue("-2147483648"))),
null));
partitionInfo = new RangePartitionInfo(partitionColumns);
for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) {
@ -99,13 +101,13 @@ public class RangePartitionInfoTest {
partitionColumns.add(k1);
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", new PartitionKeyDesc(Lists
.newArrayList("-9223372036854775808")), null));
.newArrayList(new PartitionValue("-9223372036854775808"))), null));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p2", new PartitionKeyDesc(Lists
.newArrayList("-9223372036854775806")), null));
.newArrayList(new PartitionValue("-9223372036854775806"))), null));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p3", new PartitionKeyDesc(Lists
.newArrayList("0")), null));
.newArrayList(new PartitionValue("0"))), null));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p4", new PartitionKeyDesc(Lists
.newArrayList("9223372036854775806")), null));
.newArrayList(new PartitionValue("9223372036854775806"))), null));
partitionInfo = new RangePartitionInfo(partitionColumns);
@ -121,13 +123,13 @@ public class RangePartitionInfoTest {
partitionColumns.add(k1);
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", new PartitionKeyDesc(Lists
.newArrayList("-9223372036854775806")), null));
.newArrayList(new PartitionValue("-9223372036854775806"))), null));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p2", new PartitionKeyDesc(Lists
.newArrayList("-9223372036854775805")), null));
.newArrayList(new PartitionValue("-9223372036854775805"))), null));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p3", new PartitionKeyDesc(Lists
.newArrayList("0")), null));
.newArrayList(new PartitionValue("0"))), null));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p4", new PartitionKeyDesc(Lists
.newArrayList("9223372036854775806")), null));
.newArrayList(new PartitionValue("9223372036854775806"))), null));
partitionInfo = new RangePartitionInfo(partitionColumns);
@ -137,4 +139,209 @@ public class RangePartitionInfoTest {
}
}
/**
* PARTITION BY RANGE(`k1`, `k2`) (
* PARTITION p0 VALUES [("20190101", "100"),("20190101", "200")),
* PARTITION p1 VALUES [("20190105", "10"),("20190107", "10")),
* PARTITION p2 VALUES [("20181231", "10"),("20190101", "100")),
* PARTITION p3 VALUES [("20190105", "100"),("20190120", MAXVALUE))
* )
*/
@Test
public void testFixedRange() throws DdlException, AnalysisException {
//add columns
int columns = 2;
Column k1 = new Column("k1", new ScalarType(PrimitiveType.INT), true, null, "", "");
Column k2 = new Column("k2", new ScalarType(PrimitiveType.BIGINT), true, null, "", "");
partitionColumns.add(k1);
partitionColumns.add(k2);
//add RangePartitionDescs
PartitionKeyDesc p1 = new PartitionKeyDesc(
Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("100")),
Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("200")));
PartitionKeyDesc p2 = new PartitionKeyDesc(
Lists.newArrayList(new PartitionValue("20190105"), new PartitionValue("10")),
Lists.newArrayList(new PartitionValue("20190107"), new PartitionValue("10")));
PartitionKeyDesc p3 = new PartitionKeyDesc(
Lists.newArrayList(new PartitionValue("20181231"), new PartitionValue("10")),
Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("100")));
PartitionKeyDesc p4 = new PartitionKeyDesc(
Lists.newArrayList(new PartitionValue("20190105"), new PartitionValue("100")),
Lists.newArrayList(new PartitionValue("20190120"), PartitionValue.createMaxValue()));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", p1, null));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p2", p2, null));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p3", p3, null));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p4", p4, null));
partitionInfo = new RangePartitionInfo(partitionColumns);
for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) {
singleRangePartitionDesc.analyze(columns, null);
partitionInfo.handleNewSinglePartitionDesc(singleRangePartitionDesc, 20000L);
}
}
/**
* 失败用例 less than && fixed
* partition by range(k1,k2,k3) (
* partition p1 values less than("2019-02-01", "100", "200"),
* partition p2 values [("2020-02-01", "100", "200"), (MAXVALUE)),
* partition p3 values less than("2021-02-01")
* )
*/
@Test(expected = AnalysisException.class)
public void testFixedRange1() throws DdlException, AnalysisException {
//add columns
Column k1 = new Column("k1", new ScalarType(PrimitiveType.DATE), true, null, "", "");
Column k2 = new Column("k2", new ScalarType(PrimitiveType.INT), true, null, "", "");
Column k3 = new Column("k3", new ScalarType(PrimitiveType.INT), true, null, "", "");
partitionColumns.add(k1);
partitionColumns.add(k2);
partitionColumns.add(k3);
//add RangePartitionDescs
PartitionKeyDesc p1 = new PartitionKeyDesc(
Lists.newArrayList(new PartitionValue("2019-02-01"), new PartitionValue("100"), new PartitionValue("200")));
PartitionKeyDesc p2 = new PartitionKeyDesc(
Lists.newArrayList(new PartitionValue("2020-02-01"), new PartitionValue("100"), new PartitionValue("200")),
Lists.newArrayList(PartitionValue.createMaxValue()));
PartitionKeyDesc p3 = new PartitionKeyDesc(
Lists.newArrayList(new PartitionValue("2021-02-01")));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", p1, null));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p2", p2, null));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p3", p3, null));
partitionInfo = new RangePartitionInfo(partitionColumns);
PartitionRangeType partitionType = PartitionRangeType.INVALID;
for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) {
// check partitionType
if (partitionType == PartitionRangeType.INVALID) {
partitionType = singleRangePartitionDesc.getPartitionKeyDesc().getPartitionType();
} else if (partitionType != singleRangePartitionDesc.getPartitionKeyDesc().getPartitionType()) {
throw new AnalysisException("You can only use one of these methods to create partitions");
}
singleRangePartitionDesc.analyze(partitionColumns.size(), null);
partitionInfo.handleNewSinglePartitionDesc(singleRangePartitionDesc, 20000L);
}
}
/**
* PARTITION BY RANGE(`k1`, `k2`) (
* PARTITION p1 VALUES [(), ("20190301", "400"))
* )
*/
@Test
public void testFixedRange2() throws DdlException, AnalysisException {
//add columns
int columns = 2;
Column k1 = new Column("k1", new ScalarType(PrimitiveType.INT), true, null, "", "");
Column k2 = new Column("k2", new ScalarType(PrimitiveType.BIGINT), true, null, "", "");
partitionColumns.add(k1);
partitionColumns.add(k2);
//add RangePartitionDescs
PartitionKeyDesc p1 = new PartitionKeyDesc(new ArrayList<>(),
Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("200")));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", p1, null));
partitionInfo = new RangePartitionInfo(partitionColumns);
for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) {
singleRangePartitionDesc.analyze(columns, null);
partitionInfo.handleNewSinglePartitionDesc(singleRangePartitionDesc, 20000L);
}
}
/**
* 失败用例
* PARTITION BY RANGE(`k1`, `k2`) (
* PARTITION p1 VALUES [("20190301", "400"), ())
* )
*/
@Test (expected = AnalysisException.class)
public void testFixedRange3() throws DdlException, AnalysisException {
//add columns
int columns = 2;
Column k1 = new Column("k1", new ScalarType(PrimitiveType.INT), true, null, "", "");
Column k2 = new Column("k2", new ScalarType(PrimitiveType.BIGINT), true, null, "", "");
partitionColumns.add(k1);
partitionColumns.add(k2);
//add RangePartitionDescs
PartitionKeyDesc p1 = new PartitionKeyDesc(
Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("200")),
new ArrayList<>());
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", p1, null));
partitionInfo = new RangePartitionInfo(partitionColumns);
for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) {
singleRangePartitionDesc.analyze(columns, null);
partitionInfo.handleNewSinglePartitionDesc(singleRangePartitionDesc, 20000L);
}
}
/**
* PARTITION BY RANGE(`k1`, `k2`) (
* PARTITION p0 VALUES [("20190101", "100"),("20190201"))
* )
*/
@Test
public void testFixedRange4() throws DdlException, AnalysisException {
//add columns
int columns = 2;
Column k1 = new Column("k1", new ScalarType(PrimitiveType.INT), true, null, "", "");
Column k2 = new Column("k2", new ScalarType(PrimitiveType.BIGINT), true, null, "", "");
partitionColumns.add(k1);
partitionColumns.add(k2);
//add RangePartitionDescs
PartitionKeyDesc p1 = new PartitionKeyDesc(
Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("100")),
Lists.newArrayList(new PartitionValue("20190201")));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", p1, null));
partitionInfo = new RangePartitionInfo(partitionColumns);
for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) {
singleRangePartitionDesc.analyze(columns, null);
partitionInfo.handleNewSinglePartitionDesc(singleRangePartitionDesc, 20000L);
}
}
/**
* 失败用例
* PARTITION BY RANGE(`k1`, `k2`) (
* PARTITION p0 VALUES [("20190101", "100"),("20190101", "100"))
* )
*/
@Test (expected = DdlException.class)
public void testFixedRange5() throws DdlException, AnalysisException {
//add columns
int columns = 2;
Column k1 = new Column("k1", new ScalarType(PrimitiveType.INT), true, null, "", "");
Column k2 = new Column("k2", new ScalarType(PrimitiveType.BIGINT), true, null, "", "");
partitionColumns.add(k1);
partitionColumns.add(k2);
//add RangePartitionDescs
PartitionKeyDesc p1 = new PartitionKeyDesc(
Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("100")),
Lists.newArrayList(new PartitionValue("20190101"), new PartitionValue("100")));
singleRangePartitionDescs.add(new SingleRangePartitionDesc(false, "p1", p1, null));
partitionInfo = new RangePartitionInfo(partitionColumns);
for (SingleRangePartitionDesc singleRangePartitionDesc : singleRangePartitionDescs) {
singleRangePartitionDesc.analyze(columns, null);
partitionInfo.handleNewSinglePartitionDesc(singleRangePartitionDesc, 20000L);
}
}
}

10
fe/src/test/java/org/apache/doris/es/EsStateStoreTest.java
View File

@ -22,6 +22,7 @@ import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.catalog.Catalog;
import org.apache.doris.catalog.CatalogTestUtil;
import org.apache.doris.catalog.EsTable;
@ -41,7 +42,6 @@ import com.google.common.collect.Range;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Ignore;
import org.junit.Test;
import java.io.BufferedReader;
@ -119,16 +119,16 @@ public class EsStateStoreTest {
assertEquals(5, esIndexState1.getShardRoutings().size());
assertEquals("index1", esIndexState1.getIndexName());
PartitionKey lowKey = PartitionKey.createInfinityPartitionKey(definedPartInfo.getPartitionColumns(), false);
PartitionKey upperKey = PartitionKey.createPartitionKey(Lists.newArrayList("2018-10-01"),
PartitionKey upperKey = PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("2018-10-01")),
definedPartInfo.getPartitionColumns());
Range<PartitionKey> newRange = Range.closedOpen(lowKey, upperKey);
assertEquals(newRange, part0);
Range<PartitionKey> part1 = rangeMap.get(new Long(1));
EsIndexState esIndexState2 = esTableState.getIndexState(1);
assertEquals("index2", esIndexState2.getIndexName());
lowKey = PartitionKey.createPartitionKey(Lists.newArrayList("2018-10-01"),
lowKey = PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("2018-10-01")),
definedPartInfo.getPartitionColumns());
upperKey = PartitionKey.createPartitionKey(Lists.newArrayList("2018-10-02"),
upperKey = PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("2018-10-02")),
definedPartInfo.getPartitionColumns());
newRange = Range.closedOpen(lowKey, upperKey);
assertEquals(newRange, part1);
@ -171,7 +171,7 @@ public class EsStateStoreTest {
assertEquals(5, esIndexState1.getShardRoutings().size());
assertEquals("index1", esIndexState1.getIndexName());
PartitionKey lowKey = PartitionKey.createInfinityPartitionKey(definedPartInfo.getPartitionColumns(), false);
PartitionKey upperKey = PartitionKey.createPartitionKey(Lists.newArrayList("2018-10-01"),
PartitionKey upperKey = PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("2018-10-01")),
definedPartInfo.getPartitionColumns());
Range<PartitionKey> newRange = Range.closedOpen(lowKey, upperKey);
assertEquals(newRange, part0);

3
fe/src/test/java/org/apache/doris/planner/OlapTableSinkTest.java
View File

@ -18,6 +18,7 @@
package org.apache.doris.planner;
import org.apache.doris.analysis.DescriptorTable;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.analysis.SlotDescriptor;
import org.apache.doris.analysis.TupleDescriptor;
import org.apache.doris.catalog.Column;
@ -115,7 +116,7 @@ public class OlapTableSinkTest {
2, Lists.newArrayList(new Column("k1", PrimitiveType.BIGINT)));
Column partKey = new Column("k2", PrimitiveType.VARCHAR);
PartitionKey key = PartitionKey.createPartitionKey(Lists.newArrayList("123"), Lists.newArrayList(partKey));
PartitionKey key = PartitionKey.createPartitionKey(Lists.newArrayList(new PartitionValue("123")), Lists.newArrayList(partKey));
Partition p1 = new Partition(1, "p1", index, distInfo);
Partition p2 = new Partition(2, "p2", index, distInfo);

3
fe/src/test/java/org/apache/doris/task/AgentTaskTest.java
View File

@ -17,6 +17,7 @@
package org.apache.doris.task;
import org.apache.doris.analysis.PartitionValue;
import org.apache.doris.catalog.AggregateType;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.KeysType;
@ -98,7 +99,7 @@ public class AgentTaskTest {
columns.add(new Column("v1", ScalarType.createType(PrimitiveType.INT), false, AggregateType.SUM, "1", ""));
PartitionKey pk1 = PartitionKey.createInfinityPartitionKey(Arrays.asList(columns.get(0)), false);
PartitionKey pk2 = PartitionKey.createPartitionKey(Arrays.asList("10"), Arrays.asList(columns.get(0)));
PartitionKey pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("10")), Arrays.asList(columns.get(0)));
range1 = Range.closedOpen(pk1, pk2);
PartitionKey pk3 = PartitionKey.createInfinityPartitionKey(Arrays.asList(columns.get(0)), true);