Commit Graph

8 Commits

Author SHA1 Message Date
6444187908 Fix Bug : Load parquet data during the upgrade may result in data errors (#2556) 2019-12-24 23:27:33 +08:00
35b2800542 Keep num_of_columns_from_file incompatibile with 0.10 protocol (#2187)
After checking, I found that broker load in 0.11 added num_of_columns_from_file parameter in thrift. This parameter does not consider compatibility in BE.
So broker load could cause BE crashed during the upgrade
2019-11-13 22:04:15 +08:00
cd2b8373c2 Fix Stream load double NumberTotalRows (#1664) 2019-08-19 12:23:43 +08:00
ba6d728f26 Enable parsing columns from file path for Broker Load (#1582) (#1635)
Currently, we do not support parsing encoded/compressed columns in file path, eg: extract column k1 from file path /path/to/dir/k1=1/xxx.csv

This patch is able to parse columns from file path like in Spark(Partition Discovery).

This patch parse partition columns at BrokerScanNode.java and save parsing result of each file path as a property of TBrokerRangeDesc, then the broker reader of BE can read the value of specified partition column.
2019-08-19 09:39:21 +08:00
9402456f5b Fix parquet directory have empty file (#1593) 2019-08-07 15:08:22 +08:00
97718a35a2 Do not get file size in Broker openReader() method (#1560)
The file is already got when listing files.
Get file size in openReader() again is unnecessary and inefficient.
2019-07-29 23:05:01 +08:00
b9c79d4b1b Fix importing non-parquet format file causing be crash (#1454) 2019-07-11 16:04:36 +08:00
7eab12a40e Support reading Parquet file when loading data (#1173) 2019-07-01 18:39:27 +08:00