database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
Files
00a571854195b89a4dff12e5260dd8a2d8cb0b8e
doris/regression-test/data/load_p0/stream_load
History
hui lai c66df8d9e6 [branch-2.1](load) fix no error url if no partition can be found (#36831) (#37401) 
## Proposed changes

pick #36831

before
```
Stream load result: {
    "TxnId": 2014,
    "Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]Encountered unqualified data, stop processing",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1669,
    "LoadTimeMs": 58,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 10,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 47,
    "CommitAndPublishTimeMs": 0
}
```

after
```
Stream load result: {
    "TxnId": 2014,
    "Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]too many filtered rows",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 1,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1669,
    "LoadTimeMs": 58,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 10,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 47,
    "CommitAndPublishTimeMs": 0,
    "ErrorURL": "http://XXXX:8040/api/_load_error_log?file=__shard_4/error_log_insert_stmt_c6461270125a615b-2873833fb48d56a3_c6461270125a615b_2873833fb48d56a3"
}
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-08 10:41:33 +08:00
..
load_json_parse_failed
[bug](fix) heap use after free when json parse failed (#33955)
2024-04-22 22:33:24 +08:00
all_types.csv
…
all_types.csv.gz
…
array_malformat.csv
…
array_normal.csv
[FIX](complextype) fix empty quote with complex type (#27942)
2023-12-05 12:25:26 +08:00
array_test.parquet
[fix](array)(parquet) fix be core dump due to load from parquet file containing array types (#17298)
2023-03-06 15:18:42 +08:00
basic_array_data_by_line.json
[enhancement](regression-test) add stream load json case (#25168)
2023-10-09 16:40:39 +08:00
basic_array_data_timezone.csv
[Improvement](regression-test) add stream load case (#24396)
2023-09-26 15:35:19 +08:00
basic_array_data_with_errors.csv
[Improvement](regression-test) add stream load case (#24396)
2023-09-26 15:35:19 +08:00
basic_array_data.csv
[Improvement](regression-test) add stream load case (#24396)
2023-09-26 15:35:19 +08:00
basic_array_data.csv.bz2
[Improvement](regression-test) add http_stream case (#24930)
2023-09-27 09:55:52 +08:00
basic_array_data.csv.deflate
[Improvement](regression-test) add stream load case (#24396)
2023-09-26 15:35:19 +08:00
basic_array_data.csv.gz
[Improvement](regression-test) add http_stream case (#24930)
2023-09-27 09:55:52 +08:00
basic_array_data.csv.lz4
[Improvement](regression-test) add stream load case (#24396)
2023-09-26 15:35:19 +08:00
basic_array_data.csv.lzo
[Improvement](regression-test) add stream load case (#24396)
2023-09-26 15:35:19 +08:00
basic_array_data.json
[enhancement](regression-test) add stream load json case (#25168)
2023-10-09 16:40:39 +08:00
basic_data_by_line.json
[enhancement](regression-test) add stream load json case (#25168)
2023-10-09 16:40:39 +08:00
basic_data_by_line.json.bz2
[feature](load) support compressed JSON format data for broker load (#30809)
2024-04-10 14:20:53 +08:00
basic_data_by_line.json.gz
[feature](load) support compressed JSON format data for broker load (#30809)
2024-04-10 14:20:53 +08:00
basic_data_by_line.json.lz4
[feature](load) support compressed JSON format data for broker load (#30809)
2024-04-10 14:20:53 +08:00
basic_data_timezone.csv
[Improvement](regression-test) add stream load case (#24396)
2023-09-26 15:35:19 +08:00
basic_data_with_errors.csv
[Improvement](regression-test) add stream load case (#24396)
2023-09-26 15:35:19 +08:00
basic_data.csv
[Improvement](regression-test) add stream load case (#24396)
2023-09-26 15:35:19 +08:00
basic_data.csv.bz2
[Improvement](regression-test) add stream load case (#24396)
2023-09-26 15:35:19 +08:00
basic_data.csv.deflate
[Improvement](regression-test) add stream load case (#24396)
2023-09-26 15:35:19 +08:00
basic_data.csv.gz
[Improvement](regression-test) add stream load case (#24396)
2023-09-26 15:35:19 +08:00
basic_data.csv.lz4
[Improvement](regression-test) add stream load case (#24396)
2023-09-26 15:35:19 +08:00
basic_data.csv.lzo
[Improvement](regression-test) add stream load case (#24396)
2023-09-26 15:35:19 +08:00
basic_data.json
[enhancement](regression-test) add stream load json case (#25168)
2023-10-09 16:40:39 +08:00
bitmap_hll.csv.bz2
…
case_sensitive_json.json
[Enhencement](stream load) add some regression test for json format streamload (#17520)
2023-03-12 20:13:07 +08:00
csv_with_double_quotes.csv
[feature](load) stream load trim double quotes for csv (#15241)
2022-12-26 11:45:54 +08:00
csv_with_none_utf8_data.csv
[opt](error msg) Make data codec error clearly when load csv data can't display (#25540)
2023-10-18 16:12:22 +08:00
csv_with_skip_lines.csv
[feature](Load)Suppot skip specific lines number for csv stream load (#16055)
2023-02-01 20:42:43 +08:00
enclose_empty_values.csv
[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)
2023-08-15 09:23:53 +08:00
enclose_incomplete.csv
[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)
2023-08-15 09:23:53 +08:00
enclose_multi_char_delimiter.csv
[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)
2023-08-15 09:23:53 +08:00
enclose_normal.csv
[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)
2023-08-15 09:23:53 +08:00
enclose_not_trim_quotes.csv
[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)
2023-08-15 09:23:53 +08:00
enclose_with_escape.csv
[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)
2023-08-15 09:23:53 +08:00
enclose_without_escape.csv
[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)
2023-08-15 09:23:53 +08:00
enclose_wrong_position.csv
[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)
2023-08-15 09:23:53 +08:00
invalid_json_array1.json
(Enhancement)[load-json] support simdjson in new json reader (#16903)
2023-02-21 11:31:00 +08:00
invalid_json_array2.json
(Enhancement)[load-json] support simdjson in new json reader (#16903)
2023-02-21 11:31:00 +08:00
invalid_json_array3.json
(Enhancement)[load-json] support simdjson in new json reader (#16903)
2023-02-21 11:31:00 +08:00
invalid_json_array4.json
(Enhancement)[load-json] support simdjson in new json reader (#16903)
2023-02-21 11:31:00 +08:00
invalid_json_array.json
(Enhancement)[load-json] support simdjson in new json reader (#16903)
2023-02-21 11:31:00 +08:00
invalid_json.json
(Enhancement)[load-json] support simdjson in new json reader (#16903)
2023-02-21 11:31:00 +08:00
invalid_nest_json1.json
(Enhancement)[load-json] support simdjson in new json reader (#16903)
2023-02-21 11:31:00 +08:00
invalid_nest_json2.json
[chore](config) turnoff fuzzy for enable_simdjson_reader (#25521)
2023-10-17 18:42:11 +08:00
invalid_nest_json3.json
[Optimize](simd json reader) Cached search results for previous row (keyed as index in JSON object) - used as a hint. (#17124)
2023-02-27 10:39:22 +08:00
invalid_nest_json_array1.json
(Enhancement)[load-json] support simdjson in new json reader (#16903)
2023-02-21 11:31:00 +08:00
invalid_nest_json_array2.json
(Enhancement)[load-json] support simdjson in new json reader (#16903)
2023-02-21 11:31:00 +08:00
invalid_nest_json_array3.json
(Enhancement)[load-json] support simdjson in new json reader (#16903)
2023-02-21 11:31:00 +08:00
invalid_nest_json_array.json
(Enhancement)[load-json] support simdjson in new json reader (#16903)
2023-02-21 11:31:00 +08:00
json_column_match.json
…
load_json_null_to_nullable.out
…
load_json_with_jsonpath.out
…
load_nullable_to_not_nullable.out
…
map_2_rows.json
[FIX](Map) fix map compaction error (#17795)
2023-03-16 13:54:01 +08:00
map_4093_rows.json
[FIX](Map) fix map compaction error (#17795)
2023-03-16 13:54:01 +08:00
map_char_test.csv
[FIX](collectiontype) fix shrink char column in map/struct (#25725)
2023-10-25 17:16:13 +08:00
multi_line_json.json
…
nest_json_array.json
(Enhancement)[load-json] support simdjson in new json reader (#16903)
2023-02-21 11:31:00 +08:00
nest_json.json
…
num_as_string.json
[regression-test](num_as_string) test num_as_string (#26842)
2023-11-13 21:48:13 +08:00
simple_json2_lack_one_column.json
…
simple_json2.json
(Enhancement)[load-json] support simdjson in new json reader (#16903)
2023-02-21 11:31:00 +08:00
simple_json_bom.json
[Improve](simdjson reader) support UTF-8 unicode (with BOM) (#18585)
2023-04-13 21:58:44 +08:00
simple_json.json
…
simple_object_json.json
…
special_seperator.csv
[Fix](Csv-Reader)Fix the issue of BE core dump caused by improper configuration of column_seperator and line_delimiter. (#33693)
2024-04-20 20:06:48 +08:00
struct_malformat.csv
[feature](struct-type) support csv format stream load for struct type (#17143)
2023-03-01 15:48:48 +08:00
struct_normal.csv
[FIX](complextype) fix empty quote with complex type (#27942)
2023-12-05 12:25:26 +08:00
test_array_split_string.csv
[FIX](load)fix load with split-by-string (#33713)
2024-04-17 23:42:14 +08:00
test_array_with_func_load.out
[FIX](load)fix load with split-by-string (#33713)
2024-04-17 23:42:14 +08:00
test_auth.csv
[Bug](streamload) fix inconsistent load result of be and fe (#20950)
2023-06-21 18:12:51 +08:00
test_cast1.csv
[test](cast) add test for stream load cast (#33189)
2024-04-10 15:26:09 +08:00
test_cast1.json
[test](cast) add test for stream load cast (#33189)
2024-04-10 15:26:09 +08:00
test_cast2.csv
[test](cast) add test for stream load cast (#33189)
2024-04-10 15:26:09 +08:00
test_cast2.json
[test](cast) add test for stream load cast (#33189)
2024-04-10 15:26:09 +08:00
test_cast3.csv
[test](cast) add test for stream load cast (#33189)
2024-04-10 15:26:09 +08:00
test_cast3.json
[test](cast) add test for stream load cast (#33189)
2024-04-10 15:26:09 +08:00
test_char.json
…
test_chunked_transfer.csv
[feature](streamload) support HTTP request with chunked transfer (#26520)
2023-11-08 10:07:05 +08:00
test_chunked_transfer.json
[improvement](regression-test) add chunked transfer json test (#26902)
2023-11-14 10:31:30 +08:00
test_compress_type.out
[improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library (#30573)
2024-02-05 22:00:24 +08:00
test_compress.csv.bz2
[feature](insert) Support group commit stream load (#24304)
2023-09-26 20:57:02 +08:00
test_compress.csv.gz
[feature](insert) Support group commit stream load (#24304)
2023-09-26 20:57:02 +08:00
test_compress.csv.lz4
[feature](insert) Support group commit stream load (#24304)
2023-09-26 20:57:02 +08:00
test_csv_special_seperator.out
[Fix](Csv-Reader)Fix the issue of BE core dump caused by improper configuration of column_seperator and line_delimiter. (#33693)
2024-04-20 20:06:48 +08:00
test_csv_split_line1.csv
[fix](csv-reader)fix bug of multi-char delimiter in csv reader
2023-08-23 15:19:13 +08:00
test_csv_split_line2.csv
[fix](csv_reader)Fixed bug when parsing multi-character delimiters. (#24572)
2023-09-20 12:41:35 +08:00
test_csv_split_line3.csv
[fix](csv_reader)Fixed bug when parsing multi-character delimiters. (#24572)
2023-09-20 12:41:35 +08:00
test_csv_split_line4.csv
[fix](csv_reader)Fixed bug when parsing multi-character delimiters. (#24572)
2023-09-20 12:41:35 +08:00
test_csv_split_line.out
[fix](csv_reader)Fixed bug when parsing multi-character delimiters. (#24572)
2023-09-20 12:41:35 +08:00
test_csv_with_double_quotes.out
[fix](csv_reader) fix trim_double_quotes behavior change (#27882)
2023-12-03 22:57:55 +08:00
test_csv_with_enclose_and_escape.out
[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)
2023-08-15 09:23:53 +08:00
test_csv_with_skip_lines.out
[feature](Load)Suppot skip specific lines number for csv stream load (#16055)
2023-02-01 20:42:43 +08:00
test_decimal.csv
[Fix](mutli-catalog) Use decimal v3 type to fix decimal loss issue in multi-catalog module. (#18835)
2023-04-20 11:02:53 +08:00
test_decimal.json
[Fix](mutli-catalog) Use decimal v3 type to fix decimal loss issue in multi-catalog module. (#18835)
2023-04-20 11:02:53 +08:00
test_decimal.orc
[Fix](mutli-catalog) Use decimal v3 type to fix decimal loss issue in multi-catalog module. (#18835)
2023-04-20 11:02:53 +08:00
test_decimal.parquet
[Fix](mutli-catalog) Use decimal v3 type to fix decimal loss issue in multi-catalog module. (#18835)
2023-04-20 11:02:53 +08:00
test_default_value.csv
[regression-test](load) add streamload default value test (#21536)
2023-07-06 10:14:13 +08:00
test_double_quotes_with_enclose.csv
[fix](csv_reader) fix trim_double_quotes behavior change (#27882)
2023-12-03 22:57:55 +08:00
test_empty_file.csv
[test](streamload) add load empty file regression test (#34110)
2024-04-26 07:42:09 +08:00
test_error_url_1.csv
[branch-2.1](load) fix no error url if no partition can be found (#36831) (#37401)
2024-07-08 10:41:33 +08:00
test_error_url.csv
[fix](pipeline-load) fix no error url when data quality error and total rows is negative (#34072) (#34204)
2024-04-27 18:19:08 +08:00
test_group_commit_and_wal_back_pressure.csv.gz
[Fix](core) Fix wal space back pressure core and add regression test (#27311)
2023-11-27 15:10:26 +08:00
test_group_commit_and_wal_back_pressure.out
[Fix](core) Fix wal space back pressure core and add regression test (#27311)
2023-11-27 15:10:26 +08:00
test_group_commit_stream_load.out
[feature](insert) Support group commit stream load (#24304)
2023-09-26 20:57:02 +08:00
test_group_commit_wal_limit.csv.gz
[Enhancement](group commit) Add regression case for wal limit (#27949)
2023-12-06 14:23:50 +08:00
test_hdfs_json_load.out
[fix](hdfs)read 'fs.defaultFS' from core-site.xml for hdfs load which has no default fs (#34217) (#34372)
2024-05-01 00:31:49 +08:00
test_immutable_partition.csv
[Feature](load) Support mutable property for partition (#16036)
2023-02-18 23:09:34 +08:00
test_include_where_expr.json
[Fix](Plan)StreamLoad cannot be parsed correctly when it contains complex where conditions (#23874)
2023-09-05 11:26:59 +08:00
test_input_long_than_schema.csv
[fix](load) fix loaded rows error (#28061)
2023-12-07 14:43:51 +08:00
test_json_error.json
[improve](json_reader) add prompt when all fields is null (#27630)
2023-11-29 18:26:42 +08:00
test_json_load_default_value.out
[fix](load)Fix wrong default value for char and varchar of reading json data (#22626)
2023-08-05 12:47:27 +08:00
test_json_load.out
[regression-test](load) add something like $.tag.[a.b] key's json case (#35134)
2024-05-31 22:45:09 +08:00
test_line_delimiter.csv
…
test_load_block_to_single_tablet.csv
[bug](random distribution) fix data loss and incorrect in random distribution table #33962
2024-04-24 17:13:50 +08:00
test_load_block_to_single_tablet.out
[bug](random distribution) fix data loss and incorrect in random distribution table #33962
2024-04-24 17:13:50 +08:00
test_load_to_single_tablet.json
[opt](load)change load_to_single_tablet tablet search algorithm from random to round-robin (#25256)
2023-10-16 16:43:25 +08:00
test_load_with_decimal.out
[Fix](mutli-catalog) Use decimal v3 type to fix decimal loss issue in multi-catalog module. (#18835)
2023-04-20 11:02:53 +08:00
test_load_with_jsonpath.json
…
test_load_with_map_nested_array.out
[FIX](regresstest) fix test_map_nested_array csv file for id(#27105)
2023-11-17 04:20:02 -06:00
test_malformed_json_with_path.json
[Bug](json reader) object should stop processing when encounter error (#31159)
2024-02-21 13:53:32 +08:00
test_map_load_and_compaction.out
[Improve](complex-type) Support Count(complexType) (#17868)
2023-03-30 15:43:32 +08:00
test_map_load_and_function.out
[Fix](Serde) Fix content displayed by complex types in MySQL Client (#25946)
2023-11-01 23:48:55 +08:00
test_map_nested_array.csv
[FIX](regresstest) fix test_map_nested_array csv file for id(#27105)
2023-11-17 04:20:02 -06:00
test_map.csv
[Feature](map-type) Support stream load and fix some bugs for map type (#16776)
2023-02-19 15:11:54 +08:00
test_orc_case.orc
[Fix](load)Use lower case for load column names. #16422
2023-02-07 09:18:37 +08:00
test_parquet_case.parquet
[Fix](load)Use lower case for load column names. #16422
2023-02-07 09:18:37 +08:00
test_parquet_orc_case.out
[Conf](decimalv3) enable decimalv3 by default
2023-05-29 15:38:31 +08:00
test_pipeline_load.out
[case](regression) Test enable pipeline load (#28172)
2023-12-28 10:49:19 +08:00
test_publish_timeout.out
[case](regression) streamload publish timeout (#29457)
2024-01-07 19:50:16 +08:00
test_single_quote.csv
[fix](csv_reader) fix trim_double_quotes behavior change (#27882)
2023-12-03 22:57:55 +08:00
test_special_key_json.json
[regression-test](load) add something like $.tag.[a.b] key's json case (#35134)
2024-05-31 22:45:09 +08:00
test_stream_load1.csv
[feature](insert) Support group commit stream load (#24304)
2023-09-26 20:57:02 +08:00
test_stream_load2.csv
[feature](insert) Support group commit stream load (#24304)
2023-09-26 20:57:02 +08:00
test_stream_load3.csv
[feature](insert) Support group commit stream load (#24304)
2023-09-26 20:57:02 +08:00
test_stream_load_cast.out
[test](cast) add test for stream load cast (#33189)
2024-04-10 15:26:09 +08:00
test_stream_load_data_model.csv
[Enhancement](Load) Stream Load using SQL (#22509)
2023-08-08 13:49:04 +08:00
test_stream_load_empty_file.out
[test](streamload) add load empty file regression test (#34110)
2024-04-26 07:42:09 +08:00
test_stream_load_err_log_limit.csv
[Enhancement](load) Limit the number of incorrect data drops and add documents (#27727)
2023-12-22 10:43:18 +08:00
test_stream_load_err_log_limit.out
[Enhancement](load) Limit the number of incorrect data drops and add documents (#27727)
2023-12-22 10:43:18 +08:00
test_stream_load_hll_type.csv
[improvement](stream load)(cherry-pick) support hll_from_base64 for stream load column mapping (#36819)
2024-06-26 20:12:40 +08:00
test_stream_load_move_memtable.out
[feature](serde) support presto compatible output format (#37039) (#37253)
2024-07-04 13:56:05 +08:00
test_stream_load_multiple_times.csv
[Enhancement](Load) Stream Load using SQL (#22509)
2023-08-08 13:49:04 +08:00
test_stream_load_new_change_column_order.csv
[Enhancement](Load) Stream Load using SQL (#22509)
2023-08-08 13:49:04 +08:00
test_stream_load_new_column_number_mismatch.csv
[Enhancement](Load) Stream Load using SQL (#22509)
2023-08-08 13:49:04 +08:00
test_stream_load_new_column_separator.csv
[Enhancement](Load) Stream Load using SQL (#22509)
2023-08-08 13:49:04 +08:00
test_stream_load_new_column_type.csv
[Enhancement](Load) Stream Load using SQL (#22509)
2023-08-08 13:49:04 +08:00
test_stream_load_new_current_timestamp.csv
[Enhancement](Load) Stream Load using SQL (#22509)
2023-08-08 13:49:04 +08:00
test_stream_load_new_default_value.csv
[Enhancement](Load) Stream Load using SQL (#22509)
2023-08-08 13:49:04 +08:00
test_stream_load_new_function.csv
[Enhancement](Load) Stream Load using SQL (#22509)
2023-08-08 13:49:04 +08:00
test_stream_load_new_line_delimiter.csv
[Enhancement](Load) Stream Load using SQL (#22509)
2023-08-08 13:49:04 +08:00
test_stream_load_new_move_memtable.out
[feature](move-memtable)[7/7] add regression tests (#23515)
2023-08-26 17:52:10 +08:00
test_stream_load_new.out
[improvement](stream load)(cherry-pick) support hll_from_base64 for stream load column mapping (#36819)
2024-06-26 20:12:40 +08:00
test_stream_load_properties.out
[fix](nereids)change the decimal's precision and scale for cast(xx as decimal) (#36540)
2024-06-20 17:46:11 +08:00
test_stream_load.out
[feature](serde) support presto compatible output format (#37039) (#37253)
2024-07-04 13:56:05 +08:00
test_strict_mode_fail.csv
…
test_strict_mode.csv
…
test_time.data
…
test_two_phase_commit.csv
[Feature](StreamLoad2PC) Support commit and abort streamload2PC by label (#24613)
2023-09-25 22:21:27 +08:00
test_txt_special_delimiter.csv
[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)
2023-08-15 09:23:53 +08:00
test_txt_special_delimiter.out
…
trim_double_quotes.csv
[regression-test](TRIM_DOUBLE_QUOTES) add case for TRIM_DOUBLE_QUOTES (#26998)
2023-11-14 23:52:40 +08:00
unique_key_with_delete.csv
…
unique_key.csv.lz4
…
with_jsonb.json
[Improve](simdjson) put unescaped string value after parsed (#21866)
2023-07-20 10:33:17 +08:00
with_jsonpath.json
[Fix](json reader) fix rapidjson array->PushBack may take ownership… (#21988)
2023-07-21 17:02:01 +08:00