database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[Bug]Parquet map/list/struct structure recognize (#4968)

Browse Source 
When a parquet file contains a `Map/List/Struct` structure, Doris can not recognize the column correctly,
and throws exception 'Invalid column: xxxx', that means Doris can not find the column.
The `Map` structure will be recognized into two columns: `key and value`.
The follow is the schema of a parquet file recognized by Doris. This patch tries to solve this problem.
This commit is contained in:
xinghuayu007
2020-11-28 09:56:29 +08:00
committed by GitHub
parent cb749ce51d
commit 2331ce10f1
1 changed files with 5 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
be/src/exec/parquet_reader.cpp
View File

@ -68,7 +68,11 @@ Status ParquetReaderWrap::init_parquet_reader(const std::vector<SlotDescriptor*>
auto *schemaDescriptor = _file_metadata->schema();
for (int i = 0; i < _file_metadata->num_columns(); ++i) {
// Get the Column Reader for the boolean column
_map_column.emplace(schemaDescriptor->Column(i)->name(), i);
if (schemaDescriptor->Column(i)->max_definition_level() > 1) {
_map_column.emplace(schemaDescriptor->Column(i)->path()->ToDotVector()[0], i);
} else {
_map_column.emplace(schemaDescriptor->Column(i)->name(), i);
}
}
_timezone = timezone;