Commit Graph

17 Commits

Author SHA1 Message Date
569d0bb3af Replace all remaining boost::split() with strings::split() (#2302) 2019-11-26 22:22:14 +08:00
cd5cfea5cc Encapsulate HLL logic (#1756) 2019-09-09 15:52:10 +08:00
9d03ba236b Uniform Status (#1317) 2019-06-14 23:38:31 +08:00
79ab7f4413 Change label of broker load txn (#1134)
* Change label of broker load txn

1. put broker load label into txn label
2. fix the bug of `label is already used`
3. fix partition error of new broker load

* Fix count error in mini load and broker load

There are three params (num_rows_load_total, num_rows_load_filtered, num_rows_load_unselected) which are used to count dpp.norm.ALL and dpp.abnorm.ALL.
num_rows_load_total is the number rows of source file.
num_rows_load_unselected is the not satisfied (where conjuncts) rows of num_rows_load_total
num_rows_load_filtered is the rows (quality not good enough) of (num_rows_load_total-num_rows_load_unselected)
2019-05-10 16:53:46 +08:00
a51ce03595 Enhance the usability of Load operation (#490)
1. Add broker load error hub
A broker load error hub will collect error messages in load process and saves them as a file to the specified remote storage via broker. In case that in broker/min/streaming load process, user may not be able to access the error log file in Backend directly.
We also add a new header option: 'enable_hub' in streaming load request, and default is false. Because if we enable the broker load error hub, it will significantly slow down the processing speed of streaming load, due to the visit of remote storage via broker. So use can disable the error load hub using this header option, to avoid slowing down the load speed.

2. Show load error logs by using SHOW LOAD WARNINGS stmt
We also provide a more easy way to get load error logs. We implement 'SHOW LOAD WARNINGS ON 'url'' stmt to show load error logs directly. The 'url' in stmt is provided in 'SHOW  LOAD' stmt.
eg:
show load warnings on "http://192.168.1.1:8040/api/_load_error_log?file=__shard_2/error_log_xxx";

3. Support now() function in broker load
User can mapping a column to now() in broker load stmt, which means this column will be filled with time when the ETL started.

4. Support more types of wildcard in broker load
Currently, we only support wildcard '*' to match the file names. wildcard like '/path/to/20190[1-4]*' is not support.
2019-01-03 19:07:27 +08:00
74cc5c5404 Write summary line in load error file anyway (#425)
Summary line should be wrote in spite of the limit error number
2018-12-13 12:35:19 +08:00
530bdec020 Fix bug that null value will be loaded to non-nullable column (#401)
* Fix bug that null value will be loaded to non-nullable column

* Optimize performance
2018-12-06 19:55:55 +08:00
e913e45343 Fix bug that null value will be loaded to non-nullable column (#397) 2018-12-06 10:09:34 +08:00
37b4cafe87 Change variable and namespace name in BE (#268)
Change 'palo' to 'doris'
2018-11-02 10:22:32 +08:00
2868793b6b Change license to Apache License 2.0 (#262) 2018-11-01 09:06:01 +08:00
9f7b1ea6d4 merge to 87fd4ebd9977afb1e1193429dd75c7c82caab204 (#202)
1. ix bugs in query layer.
2. remove some redundant code in BE
3. support specify multi helper node when starting FE
4. add proc 'cluster_load_statistic' to show load balance situation of Palo
2018-06-08 08:42:23 +08:00
5de798fdd6 Merge code to github (#187)
* merge to 95787f8be1fd0ff215708fb0f49997b632876586
* Bugs fixed
2018-03-23 14:04:55 +08:00
41226344ce fix csv_scan_node bug (#115) 2017-09-21 20:48:58 +08:00
6ed63ca949 fix iterator wrong compare in CsvScanNode (#106) 2017-09-20 17:36:08 +08:00
6486be64c3 fix license statement (#29)
* change picture to word

* change picture to word

* SHOW FULL TABLES WHERE Table_type != VIEW sql can not execute

* change license description
2017-08-18 19:16:23 +08:00
cc93335d29 be crashes by mini load when no hll_hash specified for hll column 2017-08-17 21:35:30 +08:00
e2311f656e baidu palo 2017-08-11 17:51:21 +08:00