database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[improvement](arrow) Avoid parse timezone for each datetime value (#10869)

Browse Source 
* [improvement](arrow) Avoid parse timezone for each datetime value

Convert arrow batch to doris block is too slow when there are datetime values.
Because we call `TimezoneUtils::find_cctz_time_zone` for each values.

After modify, the tpch-100 q1 with external table cost from 40s -> 9s

Co-authored-by: morningman <morningman@apache.org>
This commit is contained in:
Mingyu Chen
2022-07-15 21:19:36 +08:00
committed by GitHub
parent 7be2ef79ed
commit ba1c527a23
4 changed files with 29 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
be/src/vec/exec/varrow_scanner.cpp
View File

@ -271,7 +271,7 @@ Status VArrowScanner::_append_batch_to_src_block(Block* block) {
auto& column_with_type_and_name = block->get_by_name(slot_desc->col_name());
RETURN_IF_ERROR(arrow_column_to_doris_column(
array, _arrow_batch_cur_idx, column_with_type_and_name.column,
column_with_type_and_name.type, num_elements, _state->timezone()));
column_with_type_and_name.type, num_elements, _state->timezone_obj()));
}
_arrow_batch_cur_idx += num_elements;