database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

Enhance the usability of Load operation (#490)

Browse Source 
1. Add broker load error hub
A broker load error hub will collect error messages in load process and saves them as a file to the specified remote storage via broker. In case that in broker/min/streaming load process, user may not be able to access the error log file in Backend directly.
We also add a new header option: 'enable_hub' in streaming load request, and default is false. Because if we enable the broker load error hub, it will significantly slow down the processing speed of streaming load, due to the visit of remote storage via broker. So use can disable the error load hub using this header option, to avoid slowing down the load speed.

2. Show load error logs by using SHOW LOAD WARNINGS stmt
We also provide a more easy way to get load error logs. We implement 'SHOW LOAD WARNINGS ON 'url'' stmt to show load error logs directly. The 'url' in stmt is provided in 'SHOW  LOAD' stmt.
eg:
show load warnings on "http://192.168.1.1:8040/api/_load_error_log?file=__shard_2/error_log_xxx";

3. Support now() function in broker load
User can mapping a column to now() in broker load stmt, which means this column will be filled with time when the ETL started.

4. Support more types of wildcard in broker load
Currently, we only support wildcard '*' to match the file names. wildcard like '/path/to/20190[1-4]*' is not support.
This commit is contained in:
Mingyu Chen
2019-01-03 19:07:27 +08:00
committed by GitHub
parent 7057db8442
commit a51ce03595
48 changed files with 1005 additions and 372 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
be/src/exec/csv_scan_node.cpp
View File

@ -360,11 +360,14 @@ Status CsvScanNode::close(RuntimeState* state) {
return Status(error_msg.str());
}
// Summary normal line and error line number info
std::stringstream summary_msg;
summary_msg << "error line: " << _error_row_number
// only write summary line if there are error lines
if (_error_row_number > 0) {
// Summary normal line and error line number info
std::stringstream summary_msg;
summary_msg << "error line: " << _error_row_number
<< "; normal line: " << _normal_row_number;
state->append_error_msg_to_file("", summary_msg.str(), true);
state->append_error_msg_to_file("", summary_msg.str(), true);
}
return Status::OK;
}