database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
Files
f7c69ade18657bd3627decbbb841014127c6f857
doris/be/src/exec
History
Ashin Gau f7c69ade18 [feature-wip](multi-catalog) implement predicate pushdown in native OrcReader (#13453) 
# Proposed changes
Implement predicate pushdown in `OrcReader` by converting doris `ColumnValueRange` to orc `SearchArgument`.

## Remaining problems
1. Orc support `not in`, which may have effect on bloom filter. However, doris `ScanNode` has not push down `not in` to file scanner.
2. Orc support `is null`, and row range has `hasNull` identifier. However,  `_contain_null` in `ColumnValueRange` is ambiguous. `_contain_null = true` only means that the value can be nullable, not equal to null.
3. `DateTimeV2` has lost microsecond precision in `ColumnValueRange`, which may cause filtering error when a min-max value equals to the predicate value.
4. `DateTimeV1`  is not accurate enough, and only saved to seconds.
5. Orc support the predicate pushdown of `float&double` type, but doris has not push down `float&double` type for precision reason.
2022-10-20 10:07:36 +08:00
..
arrow
[feature-wip](multi-catalog) use apache orc reader to read orc file (#13404)
2022-10-18 13:47:56 +08:00
es
[Enhancement](Doe) Be query es use fe generate dsl. (#11840)
2022-08-18 10:31:17 +08:00
schema_scanner
[chore](build) Support building from source on ubuntu-22.04 (aarch64) (#12813)
2022-09-27 10:29:13 +08:00
analytic_eval_node.cpp
[enhancement](macOS M1) Support building from source on macOS (M1) (#13195)
2022-10-18 13:10:13 +08:00
analytic_eval_node.h
[refactor][be] Optimize headers (#9708)
2022-05-30 16:12:10 +08:00
assert_num_rows_node.cpp
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
assert_num_rows_node.h
[Refactor] Replace '#ifndef' with '#pragma once' (#9456)
2022-05-10 09:25:59 +08:00
base_scanner.cpp
[fix](array-type) fix the invalid format load for stream load (#12424)
2022-09-19 08:52:59 +08:00
base_scanner.h
[fix](spark-load) no need to filter row group when doing spark load (#13116)
2022-10-05 23:00:56 +08:00
blocking_join_node.cpp
[fix](memtracker) Improve memory tracking accuracy for exec nodes (#11947)
2022-08-22 08:56:05 +08:00
blocking_join_node.h
[Refactor] Replace '#ifndef' with '#pragma once' (#9456)
2022-05-10 09:25:59 +08:00
broker_scan_node.cpp
[fix](memtracker) Remove useless memory exceed check #11939
2022-08-22 08:40:19 +08:00
broker_scan_node.h
[Bug] [Build] fix clang build fail (#9323)
2022-05-02 18:04:57 +08:00
broker_scanner.cpp
[chore](fix) Fix some spell errors in be's comments. #13452
2022-10-20 08:56:01 +08:00
broker_scanner.h
[feature-wip](new-scan) Support stream load with csv in new scan framework (#13354)
2022-10-17 23:33:41 +08:00
CMakeLists.txt
[Feature](vectorized) support jdbc sink for insert into data to table (#12534)
2022-09-15 11:08:41 +08:00
cross_join_node.cpp
[bugfix](memtracker)fix exceed memory limit log (#11485)
2022-08-04 10:22:20 +08:00
cross_join_node.h
[Enhancement] Remove some unused include (#10035)
2022-06-17 10:47:25 +08:00
csv_scan_node.cpp
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
csv_scan_node.h
[Enhancement] Remove some unused include (#10035)
2022-06-17 10:47:25 +08:00
csv_scanner.cpp
[refactor] Refactoring Status static methods to format message using fmt(#9533)
2022-07-02 18:58:23 +08:00
csv_scanner.h
[Refactor] Replace '#ifndef' with '#pragma once' (#9456)
2022-05-10 09:25:59 +08:00
data_sink.cpp
[Feature](vectorized) support jdbc sink for insert into data to table (#12534)
2022-09-15 11:08:41 +08:00
data_sink.h
[enhancement](tracing) append the profile counter to trace. (#11458)
2022-08-15 21:36:38 +08:00
decompressor.cpp
[refactor] Refactoring Status static methods to format message using fmt(#9533)
2022-07-02 18:58:23 +08:00
decompressor.h
[style] clang-format all c++ code (#9305)
2022-04-29 16:14:22 +08:00
empty_set_node.cpp
[refactor] add reference path for source file from impala (#9115)
2022-04-20 12:29:57 +08:00
empty_set_node.h
[refactor] add reference path for source file from impala (#9115)
2022-04-20 12:29:57 +08:00
es_http_scan_node.cpp
[fix](exec-node) fix nullptr of runtime state (#12395)
2022-09-07 08:46:42 +08:00
es_http_scan_node.h
[improvement][vectorized] support es node predicate peel (#8174)
2022-02-26 17:02:54 +08:00
es_http_scanner.cpp
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
es_http_scanner.h
[Enhancement] Remove some unused include (#10035)
2022-06-17 10:47:25 +08:00
except_node.cpp
[bugfix](memtracker)fix exceed memory limit log (#11485)
2022-08-04 10:22:20 +08:00
except_node.h
[Refactor] Fixes some be typo part 2 (#4747)
2020-10-20 09:28:57 +08:00
exchange_node.cpp
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
exchange_node.h
[typo] Fix typos in comments (#10106)
2022-06-14 08:17:19 +08:00
exec_node.cpp
[refactor](new-scan) remove old file scan node (#13433)
2022-10-19 14:25:32 +08:00
exec_node.h
[fix](exec-node) fix nullptr of runtime state (#12395)
2022-09-07 08:46:42 +08:00
hash_join_node.cpp
[enhancement](memory) Reduce virtual memory used by PaddedPODArray (#11816)
2022-08-22 11:33:07 +08:00
hash_join_node.h
[Enhancement] Remove some unused include (#10035)
2022-06-17 10:47:25 +08:00
hash_table.cpp
[enhancement](memtracker) Improve performance of tracking real physical memory of PODArray #12168
2022-08-30 10:22:12 +08:00
hash_table.h
[enhancement](memtracker) Optimize readability of mem exceed limit error message #11877
2022-08-18 14:39:41 +08:00
intersect_node.cpp
[bugfix](memtracker)fix exceed memory limit log (#11485)
2022-08-04 10:22:20 +08:00
intersect_node.h
[Refactor] Fixes some be typo part 2 (#4747)
2020-10-20 09:28:57 +08:00
json_scanner.cpp
[chore](fix) Fix some spell errors in be's comments. #13452
2022-10-20 08:56:01 +08:00
json_scanner.h
[feature-wip](new-scan) Support stream load with csv in new scan framework (#13354)
2022-10-17 23:33:41 +08:00
line_reader.h
[Bug][Vectorized] fix core dump with HLL and some refactor of Decompressor (#8668)
2022-03-31 17:05:08 +08:00
lzo_decompressor.cpp
Fix some typos in be/. (#9681)
2022-05-19 20:55:39 +08:00
merge_node.cpp
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
merge_node.h
[Enhancement] Remove some unused include (#10035)
2022-06-17 10:47:25 +08:00
mysql_scan_node.cpp
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
mysql_scan_node.h
[Refactor] Replace '#ifndef' with '#pragma once' (#9456)
2022-05-10 09:25:59 +08:00
mysql_scanner.cpp
[feature](mysql-table) support utf8mb4 for mysql external table (#9402)
2022-05-11 09:39:23 +08:00
mysql_scanner.h
[feature](mysql-table) support utf8mb4 for mysql external table (#9402)
2022-05-11 09:39:23 +08:00
odbc_connector.cpp
[Feature](vectorized) support jdbc sink for insert into data to table (#12534)
2022-09-15 11:08:41 +08:00
odbc_connector.h
[Feature](vectorized) support jdbc sink for insert into data to table (#12534)
2022-09-15 11:08:41 +08:00
odbc_scan_node.cpp
[Bug](ODBC) fix vectorized null value error report in odbc scan node (#11420)
2022-08-02 15:44:12 +08:00
odbc_scan_node.h
[Refactor] Replace '#ifndef' with '#pragma once' (#9456)
2022-05-10 09:25:59 +08:00
olap_common.cpp
[feature-wip] Optimize Decimal type (#10794)
2022-07-14 10:50:50 +08:00
olap_common.h
[feature-wip](multi-catalog) implement predicate pushdown in native OrcReader (#13453)
2022-10-20 10:07:36 +08:00
olap_scan_node.cpp
[bugfix](scanner) olap scanner compute is wrong (#12857)
2022-09-24 09:59:59 +08:00
olap_scan_node.h
[Enhancement](runtime filter) optimize for runtime filter (#12856)
2022-10-09 14:11:03 +08:00
olap_scanner.cpp
[Enhancement](runtime filter) optimize for runtime filter (#12856)
2022-10-09 14:11:03 +08:00
olap_scanner.h
[Enhancement](runtime filter) optimize for runtime filter (#12856)
2022-10-09 14:11:03 +08:00
olap_utils.h
[Refactor](push-down predicate) Derive push-down predicate from vconjuncts (#11468)
2022-08-08 19:19:26 +08:00
orc_scanner.cpp
[enhancement](macOS M1) Support building from source on macOS (M1) (#13195)
2022-10-18 13:10:13 +08:00
orc_scanner.h
[Refactor][Bug-Fix][Load Vec] Refactor code of basescanner and vjson/vparquet/vbroker scanner (#9666)
2022-05-20 11:43:03 +08:00
parquet_scanner.cpp
[refactor](reader) refactor the interface of file reader (#12574)
2022-09-14 22:31:11 +08:00
parquet_scanner.h
[Refactor][Bug-Fix][Load Vec] Refactor code of basescanner and vjson/vparquet/vbroker scanner (#9666)
2022-05-20 11:43:03 +08:00
parquet_writer.cpp
[fix](parquet) fix write error data as parquet format. (#12864)
2022-09-26 10:41:17 +08:00
parquet_writer.h
[fix](parquet) fix write error data as parquet format. (#12864)
2022-09-26 10:41:17 +08:00
partitioned_aggregation_node.cc
[enhancement](memtracker) Improve performance of tracking real physical memory of PODArray #12168
2022-08-30 10:22:12 +08:00
partitioned_aggregation_node.h
[enhancement](memory) Remove unused reservation tracker (#11969)
2022-08-24 08:49:34 +08:00
partitioned_hash_table.cc
[Chore](clang) fix some build fail on clang15 (#12882)
2022-09-26 23:13:28 +08:00
partitioned_hash_table.h
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
partitioned_hash_table.inline.h
[Enhancement] Remove some unused include (#10035)
2022-06-17 10:47:25 +08:00
plain_binary_line_reader.cpp
[Refactor] Use file factory to replace create file reader/writer (#9505)
2022-06-08 15:07:39 +08:00
plain_binary_line_reader.h
[Feature][Insert] Add transaction for the operation of insert #6244 (#6245)
2021-07-21 10:54:11 +08:00
plain_text_line_reader.cpp
fix bug that last line of data lost for stream load when line delimiter is more than one character (#13066)
2022-10-07 16:12:05 +08:00
plain_text_line_reader.h
[feature-wip](new-scan) Support stream load with csv in new scan framework (#13354)
2022-10-17 23:33:41 +08:00
read_write_util.cpp
[CodeFormat] Clang-format cpp sources (#4965)
2020-11-28 18:36:49 +08:00
read_write_util.h
[refactor] Refactoring Status static methods to format message using fmt(#9533)
2022-07-02 18:58:23 +08:00
repeat_node.cpp
[fix] Fix the query result error caused by the grouping sets statemen… (#11316)
2022-08-01 13:52:18 +08:00
repeat_node.h
[fix] Fix the query result error caused by the grouping sets statemen… (#11316)
2022-08-01 13:52:18 +08:00
row_batch_list.h
[Enhancement] Remove some unused include (#10035)
2022-06-17 10:47:25 +08:00
scan_node.cpp
[Bug](function) fix dcheck fail on close vexpr ctx (#11908)
2022-08-19 19:11:10 +08:00
scan_node.h
[bugfix] fix coredump caused by wrong type cast of OlapScanNode (#11165)
2022-07-25 17:57:53 +08:00
schema_scan_node.cpp
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
schema_scan_node.h
[Enhancement] Remove some unused include (#10035)
2022-06-17 10:47:25 +08:00
schema_scanner.cpp
[feature-wip](parquet-reader) generate null values and NullMap for parquet column (#12115)
2022-08-29 09:30:32 +08:00
schema_scanner.h
[feature-wip](parquet-reader) decode parquet time & datetime & decimal (#11845)
2022-08-22 10:15:35 +08:00
select_node.cpp
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
select_node.h
[Enhancement] Remove some unused include (#10035)
2022-06-17 10:47:25 +08:00
set_operation_node.cpp
[fix](intersect) fix intersect query failed in row storage code (#12712)
2022-09-19 11:47:50 +08:00
set_operation_node.h
[fix](intersect) fix intersect query failed in row storage code (#12712)
2022-09-19 11:47:50 +08:00
sort_exec_exprs.cpp
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
sort_exec_exprs.h
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
spill_sort_node.cc
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
spill_sort_node.h
[Enhancement] Remove some unused include (#10035)
2022-06-17 10:47:25 +08:00
table_connector.cpp
[enhancement](macOS M1) Support building from source on macOS (M1) (#13195)
2022-10-18 13:10:13 +08:00
table_connector.h
[fix](jdbc) fix insert into date type to oracle using wrong type (#12883)
2022-10-04 21:24:33 +08:00
table_function_node.cpp
[Enhancement](table_function) table function node enhancement (#12038)
2022-08-26 10:37:15 +08:00
table_function_node.h
[fix](Lateral-View) fix outer combinator not work on non-vectorized (#9212)
2022-05-01 22:09:50 +08:00
tablet_info.cpp
[fix](BE) fix the stream load error when upgrade BE from 1.1.2 to master (#13058)
2022-10-05 12:13:26 +08:00
tablet_info.h
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
tablet_sink.cpp
[fix](streamload&sink) release and allocate memory in the same tracker (#12820)
2022-09-23 17:51:44 +08:00
tablet_sink.h
[fix](streamload&sink) release and allocate memory in the same tracker (#12820)
2022-09-23 17:51:44 +08:00
text_converter.cpp
[Enhancement] Remove some unused include (#10035)
2022-06-17 10:47:25 +08:00
text_converter.h
[fix](multi-catalog) fix the core dump on hms table (#10573)
2022-07-04 15:52:05 +08:00
text_converter.hpp
[Opt](load) Speed up the vectorized load (#12146)
2022-08-31 16:23:36 +08:00
topn_node.cpp
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
topn_node.h
[Refactor] Replace '#ifndef' with '#pragma once' (#9456)
2022-05-10 09:25:59 +08:00
union_node.cpp
[enhancement] Refactor to improve the usability of MemTracker (step2) (#10823)
2022-07-21 17:11:28 +08:00
union_node.h
[refactor] add reference path for source file from impala (#9115)
2022-04-20 12:29:57 +08:00