bd126f0679
## Design:
For now, there are two categories of types in Doris, one is for scalar types (such as int, char and etc.) and the other is for composite types (array and etc.). For the sake of performance, we can cache type info of scalar types globally (unique objects) due to the limited number of scalar types. When we consider the composite types, normally, the type info is generated in runtime (we can also use some cache strategy to speed up). The memory thereby should be reclaimed when we create type info for composite types.
There are a lots of interfaces to get the type info of a specific type. I reorganized those as the following describes.
1. `const TypeInfo* get_scalar_type_info(FieldType field_type)`
The function is used to get the type info of scalar types. Due to the cache, the caller uses the result **WITHOUT** considering the problems about memory reclaim.
2. `const TypeInfo* get_collection_type_info(FieldType sub_type)`
The function is used to get the type info of array types with just **ONE** depth. Due to the cache, the caller uses the result **WITHOUT** considering the problems about memory reclaim.
3. `TypeInfoPtr get_type_info(segment_v2::ColumnMetaPB* column_meta_pb)`
4. `TypeInfoPtr get_type_info(const TabletColumn* col)`
These functions are used to get the type info of **BOTH** scalar types and composite types. The caller should be responsible to manage the resources returned.
#### About the new type `TypeInfoPtr`
`TypeInfoPtr` is an alias type to `unique_ptr` with a custom deleter.
1. For scalar types, the deleter does nothing.
2. For composite types, the deleter reclaim the memory.
By analyzing the callers of `get_type_info`, these classes should hold TypeInfoPtr:
1. `Field`
2. `ColumnReader`
3. `DefaultValueColumnIterator`
Other classes are either constructed by the foregoing classes or hold those, so they can just use the raw pointer of `TypeInfo` directly for the sake of performance.
1. `ScalarColumnWriter` - holds `Field`
1. `ZoneMapIndexWriter` - created by `ScalarColumnWriter`, use `type_info` from the field in `ScalarColumnWriter`
1. `IndexedColumnWriter` - created by `ZoneMapIndexWriter`, only uses scalar types.
2. `BitmapIndexWriter` - created by `ScalarColumnWriter`, uses `type_info` from the field in `ScalarColumnWriter`
1. `IndexedColumnWriter` - created by `BitmapIndexWriter`, uses `type_info` in `BitmapIndexWriter` and `BitmapIndexWriter` doesn't support `ArrayType`.
3. `BloomFilterIndexWriter` - created by `ScalarColumnWriter`, uses `type_info` from the field in `ScalarColumnWriter`
1. `IndexedColumnWriter` - created by `BloomFilterIndexWriter`, only uses scalar types.
2. `IndexedColumnReader` initializes `type_info` by the field type in meta (only scalar types).
3. `ColumnVectorBatch`
1. `ZoneMapIndexReader` creates `ColumnVectorBatch`, `ColumnVectorBatch` uses `type_info` in `IndexedColumnReader`
2. `BitmapIndexReader` supports scalar types only and it creates `ColumnVectorBatch`, `ColumnVectorBatch` uses `type_info` in `BitmapIndexReader`
3. `BloomFilterIndexWriter` supports scalar types only and it creates `ColumnVectorBatch`, `ColumnVectorBatch` uses `type_info` in `BloomFilterIndexWriter`
122 lines
4.4 KiB
C++
122 lines
4.4 KiB
C++
// Licensed to the Apache Software Foundation (ASF) under one
|
|
// or more contributor license agreements. See the NOTICE file
|
|
// distributed with this work for additional information
|
|
// regarding copyright ownership. The ASF licenses this file
|
|
// to you under the Apache License, Version 2.0 (the
|
|
// "License"); you may not use this file except in compliance
|
|
// with the License. You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing,
|
|
// software distributed under the License is distributed on an
|
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
// KIND, either express or implied. See the License for the
|
|
// specific language governing permissions and limitations
|
|
// under the License.
|
|
|
|
#include <gtest/gtest.h>
|
|
|
|
#include <memory>
|
|
#include <string>
|
|
|
|
#include "olap/tablet_schema.h"
|
|
#include "olap/types.h"
|
|
#include "runtime/mem_tracker.h"
|
|
#include "runtime/string_value.h"
|
|
#include "testutil/array_utils.h"
|
|
|
|
namespace doris {
|
|
|
|
template <typename... Ts>
|
|
ColumnPB create_column_pb(const std::string& type, const Ts&... sub_column_types) {
|
|
ColumnPB column;
|
|
column.set_type(type);
|
|
column.set_aggregation("NONE");
|
|
column.set_is_nullable(true);
|
|
if (type == "ARRAY") {
|
|
column.set_length(OLAP_ARRAY_MAX_BYTES);
|
|
}
|
|
if constexpr (sizeof...(sub_column_types) > 0) {
|
|
auto sub_column = create_column_pb(sub_column_types...);
|
|
column.add_children_columns()->Swap(&sub_column);
|
|
}
|
|
return column;
|
|
}
|
|
|
|
static TypeInfoPtr get_type_info(const ColumnPB& column_pb) {
|
|
TabletColumn tablet_column;
|
|
tablet_column.init_from_pb(column_pb);
|
|
return get_type_info(&tablet_column);
|
|
}
|
|
|
|
static void test_array_parser(const ColumnPB& column_pb, const std::string& json,
|
|
const CollectionValue& expect) {
|
|
MemTracker tracker(1024 * 1024, "ArrayParserTest");
|
|
MemPool mem_pool(&tracker);
|
|
FunctionContext context;
|
|
ArrayUtils::prepare_context(context, mem_pool, column_pb);
|
|
CollectionValue actual;
|
|
auto status = ArrayUtils::create_collection_value(&actual, &context, json);
|
|
EXPECT_TRUE(status.ok());
|
|
EXPECT_TRUE(get_type_info(column_pb)->equal(&expect, &actual));
|
|
}
|
|
|
|
TEST(ArrayParserTest, TestParseIntArray) {
|
|
auto column_pb = create_column_pb("ARRAY", "INT");
|
|
test_array_parser(column_pb, "[]", CollectionValue(0));
|
|
|
|
int32_t data[] = {1, 2, 3};
|
|
int num_items = sizeof(data) / sizeof(data[0]);
|
|
CollectionValue value(data, num_items, false, nullptr);
|
|
test_array_parser(column_pb, "[1, 2, 3]", value);
|
|
|
|
bool null_signs[] = {false, true, false};
|
|
value.set_has_null(true);
|
|
value.set_null_signs(null_signs);
|
|
test_array_parser(column_pb, "[1, null, 3]", value);
|
|
}
|
|
|
|
TEST(ArrayParserTest, TestParseVarcharArray) {
|
|
auto column_pb = create_column_pb("ARRAY", "VARCHAR");
|
|
test_array_parser(column_pb, "[]", CollectionValue(0));
|
|
|
|
char data[] = {'a', 'b', 'c'};
|
|
int num_items = sizeof(data) / sizeof(data[0]);
|
|
StringValue string_values[] = {
|
|
{&data[0], 1},
|
|
{&data[1], 1},
|
|
{&data[2], 1},
|
|
};
|
|
CollectionValue value(string_values, num_items, false, nullptr);
|
|
test_array_parser(column_pb, "[\"a\", \"b\", \"c\"]", value);
|
|
|
|
bool null_signs[] = {false, true, false};
|
|
value.set_has_null(true);
|
|
value.set_null_signs(null_signs);
|
|
test_array_parser(column_pb, "[\"a\", null, \"c\"]", value);
|
|
}
|
|
|
|
TEST(ArrayParserTest, TestNestedArray) {
|
|
auto column_pb = create_column_pb("ARRAY", "ARRAY", "INT");
|
|
test_array_parser(column_pb, "[]", CollectionValue(0));
|
|
|
|
CollectionValue empty_array(0);
|
|
test_array_parser(column_pb, "[[]]", {&empty_array, 1, false, nullptr});
|
|
|
|
int data[] = {1, 0, 3};
|
|
uint32_t num_items = sizeof(data) / sizeof(data[0]);
|
|
bool null_signs[] = {false, true, false};
|
|
CollectionValue array = {data, num_items, true, null_signs};
|
|
|
|
CollectionValue array_data[] = {empty_array, array, empty_array, array};
|
|
uint32_t num_arrays = sizeof(array_data) / sizeof(array_data[0]);
|
|
test_array_parser(column_pb, "[[], [1, null, 3], [], [1, null, 3]]",
|
|
{array_data, num_arrays, false, nullptr});
|
|
bool array_null_signs[] = {false, true, true, false};
|
|
test_array_parser(column_pb, "[[], null, null, [1, null, 3]]",
|
|
{array_data, num_arrays, true, array_null_signs});
|
|
}
|
|
|
|
} // namespace doris
|