amory 338fa32303 [pick](simdjson) fix simdjson with object array when jsonroot is not empty (#38633) ...

## Proposed changes
backport: https://github.com/apache/doris/pull/38490
Issue Number: close #xxx

<!--Describe your changes.-->

2024-08-01 11:04:54 +08:00

..

load_json_parse_failed

[bug](fix) heap use after free when json parse failed (#33955)

2024-04-22 22:33:24 +08:00

all_types.csv

…

all_types.csv.gz

…

array_malformat.csv

…

array_normal.csv

[FIX](complextype) fix empty quote with complex type (#27942)

2023-12-05 12:25:26 +08:00

array_test.parquet

[fix](array)(parquet) fix be core dump due to load from parquet file containing array types (#17298)

2023-03-06 15:18:42 +08:00

basic_array_data_by_line.json

[enhancement](regression-test) add stream load json case (#25168)

2023-10-09 16:40:39 +08:00

basic_array_data_timezone.csv

[Improvement](regression-test) add stream load case (#24396)

2023-09-26 15:35:19 +08:00

basic_array_data_with_errors.csv

[Improvement](regression-test) add stream load case (#24396)

2023-09-26 15:35:19 +08:00

basic_array_data.csv

[Improvement](regression-test) add stream load case (#24396)

2023-09-26 15:35:19 +08:00

basic_array_data.csv.bz2

[Improvement](regression-test) add http_stream case (#24930)

2023-09-27 09:55:52 +08:00

basic_array_data.csv.deflate

[Improvement](regression-test) add stream load case (#24396)

2023-09-26 15:35:19 +08:00

basic_array_data.csv.gz

[Improvement](regression-test) add http_stream case (#24930)

2023-09-27 09:55:52 +08:00

basic_array_data.csv.lz4

[Improvement](regression-test) add stream load case (#24396)

2023-09-26 15:35:19 +08:00

basic_array_data.csv.lzo

[Improvement](regression-test) add stream load case (#24396)

2023-09-26 15:35:19 +08:00

basic_array_data.json

[enhancement](regression-test) add stream load json case (#25168)

2023-10-09 16:40:39 +08:00

basic_data_by_line.json

[enhancement](regression-test) add stream load json case (#25168)

2023-10-09 16:40:39 +08:00

basic_data_by_line.json.bz2

[feature](load) support compressed JSON format data for broker load (#30809)

2024-04-10 14:20:53 +08:00

basic_data_by_line.json.gz

[feature](load) support compressed JSON format data for broker load (#30809)

2024-04-10 14:20:53 +08:00

basic_data_by_line.json.lz4

[feature](load) support compressed JSON format data for broker load (#30809)

2024-04-10 14:20:53 +08:00

basic_data_timezone.csv

[Improvement](regression-test) add stream load case (#24396)

2023-09-26 15:35:19 +08:00

basic_data_with_errors.csv

[Improvement](regression-test) add stream load case (#24396)

2023-09-26 15:35:19 +08:00

basic_data.csv

[Improvement](regression-test) add stream load case (#24396)

2023-09-26 15:35:19 +08:00

basic_data.csv.bz2

[Improvement](regression-test) add stream load case (#24396)

2023-09-26 15:35:19 +08:00

basic_data.csv.deflate

[Improvement](regression-test) add stream load case (#24396)

2023-09-26 15:35:19 +08:00

basic_data.csv.gz

[Improvement](regression-test) add stream load case (#24396)

2023-09-26 15:35:19 +08:00

basic_data.csv.lz4

[Improvement](regression-test) add stream load case (#24396)

2023-09-26 15:35:19 +08:00

basic_data.csv.lzo

[Improvement](regression-test) add stream load case (#24396)

2023-09-26 15:35:19 +08:00

basic_data.json

[enhancement](regression-test) add stream load json case (#25168)

2023-10-09 16:40:39 +08:00

bitmap_hll.csv.bz2

…

case_sensitive_json.json

[Enhencement](stream load) add some regression test for json format streamload (#17520)

2023-03-12 20:13:07 +08:00

csv_with_double_quotes.csv

…

csv_with_none_utf8_data.csv

[opt](error msg) Make data codec error clearly when load csv data can't display (#25540)

2023-10-18 16:12:22 +08:00

csv_with_skip_lines.csv

…

enclose_empty_values.csv

[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)

2023-08-15 09:23:53 +08:00

enclose_incomplete.csv

[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)

2023-08-15 09:23:53 +08:00

enclose_multi_char_delimiter.csv

[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)

2023-08-15 09:23:53 +08:00

enclose_normal.csv

[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)

2023-08-15 09:23:53 +08:00

enclose_not_trim_quotes.csv

[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)

2023-08-15 09:23:53 +08:00

enclose_with_escape.csv

[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)

2023-08-15 09:23:53 +08:00

enclose_without_escape.csv

[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)

2023-08-15 09:23:53 +08:00

enclose_wrong_position.csv

[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)

2023-08-15 09:23:53 +08:00

invalid_json_array1.json

(Enhancement)[load-json] support simdjson in new json reader (#16903)

2023-02-21 11:31:00 +08:00

invalid_json_array2.json

(Enhancement)[load-json] support simdjson in new json reader (#16903)

2023-02-21 11:31:00 +08:00

invalid_json_array3.json

(Enhancement)[load-json] support simdjson in new json reader (#16903)

2023-02-21 11:31:00 +08:00

invalid_json_array4.json

(Enhancement)[load-json] support simdjson in new json reader (#16903)

2023-02-21 11:31:00 +08:00

invalid_json_array.json

(Enhancement)[load-json] support simdjson in new json reader (#16903)

2023-02-21 11:31:00 +08:00

invalid_json.json

(Enhancement)[load-json] support simdjson in new json reader (#16903)

2023-02-21 11:31:00 +08:00

invalid_nest_json1.json

(Enhancement)[load-json] support simdjson in new json reader (#16903)

2023-02-21 11:31:00 +08:00

invalid_nest_json2.json

[chore](config) turnoff fuzzy for enable_simdjson_reader (#25521)

2023-10-17 18:42:11 +08:00

invalid_nest_json3.json

[Optimize](simd json reader) Cached search results for previous row (keyed as index in JSON object) - used as a hint. (#17124)

2023-02-27 10:39:22 +08:00

invalid_nest_json_array1.json

(Enhancement)[load-json] support simdjson in new json reader (#16903)

2023-02-21 11:31:00 +08:00

invalid_nest_json_array2.json

(Enhancement)[load-json] support simdjson in new json reader (#16903)

2023-02-21 11:31:00 +08:00

invalid_nest_json_array3.json

(Enhancement)[load-json] support simdjson in new json reader (#16903)

2023-02-21 11:31:00 +08:00

invalid_nest_json_array.json

(Enhancement)[load-json] support simdjson in new json reader (#16903)

2023-02-21 11:31:00 +08:00

json_column_match.json

…

load_json_null_to_nullable.out

…

load_json_with_jsonpath.out

…

load_nullable_to_not_nullable.out

…

load_object_array_json.out

[pick](simdjson) fix simdjson with object array when jsonroot is not empty (#38633)

2024-08-01 11:04:54 +08:00

map_2_rows.json

[FIX](Map) fix map compaction error (#17795)

2023-03-16 13:54:01 +08:00

map_4093_rows.json

[FIX](Map) fix map compaction error (#17795)

2023-03-16 13:54:01 +08:00

map_char_test.csv

[FIX](collectiontype) fix shrink char column in map/struct (#25725)

2023-10-25 17:16:13 +08:00

multi_line_json.json

…

nest_json_array.json

(Enhancement)[load-json] support simdjson in new json reader (#16903)

2023-02-21 11:31:00 +08:00

nest_json.json

…

num_as_string.json

[regression-test](num_as_string) test num_as_string (#26842)

2023-11-13 21:48:13 +08:00

simple_json2_lack_one_column.json

…

simple_json2.json

(Enhancement)[load-json] support simdjson in new json reader (#16903)

2023-02-21 11:31:00 +08:00

simple_json_bom.json

[Improve](simdjson reader) support UTF-8 unicode (with BOM) (#18585)

2023-04-13 21:58:44 +08:00

simple_json.json

…

simple_object_json.json

…

special_seperator.csv

[Fix](Csv-Reader)Fix the issue of BE core dump caused by improper configuration of column_seperator and line_delimiter. (#33693)

2024-04-20 20:06:48 +08:00

struct_malformat.csv

[feature](struct-type) support csv format stream load for struct type (#17143)

2023-03-01 15:48:48 +08:00

struct_normal.csv

[FIX](complextype) fix empty quote with complex type (#27942)

2023-12-05 12:25:26 +08:00

test_array_split_string.csv

[FIX](load)fix load with split-by-string (#33713)

2024-04-17 23:42:14 +08:00

test_array_with_func_load.out

[FIX](load)fix load with split-by-string (#33713)

2024-04-17 23:42:14 +08:00

test_auth.csv

[Bug](streamload) fix inconsistent load result of be and fe (#20950)

2023-06-21 18:12:51 +08:00

test_cast1.csv

[test](cast) add test for stream load cast (#33189)

2024-04-10 15:26:09 +08:00

test_cast1.json

[test](cast) add test for stream load cast (#33189)

2024-04-10 15:26:09 +08:00

test_cast2.csv

[test](cast) add test for stream load cast (#33189)

2024-04-10 15:26:09 +08:00

test_cast2.json

[test](cast) add test for stream load cast (#33189)

2024-04-10 15:26:09 +08:00

test_cast3.csv

[test](cast) add test for stream load cast (#33189)

2024-04-10 15:26:09 +08:00

test_cast3.json

[test](cast) add test for stream load cast (#33189)

2024-04-10 15:26:09 +08:00

test_char.json

…

test_chunked_transfer.csv

[feature](streamload) support HTTP request with chunked transfer (#26520)

2023-11-08 10:07:05 +08:00

test_chunked_transfer.json

[improvement](regression-test) add chunked transfer json test (#26902)

2023-11-14 10:31:30 +08:00

test_compress_type.out

[improvement](load) Enable lzo & Remove dependency on Markus F.X.J. Oberhumer's lzo library (#30573)

2024-02-05 22:00:24 +08:00

test_compress.csv.bz2

[feature](insert) Support group commit stream load (#24304)

2023-09-26 20:57:02 +08:00

test_compress.csv.gz

[feature](insert) Support group commit stream load (#24304)

2023-09-26 20:57:02 +08:00

test_compress.csv.lz4

[feature](insert) Support group commit stream load (#24304)

2023-09-26 20:57:02 +08:00

test_csv_special_seperator.out

[Fix](Csv-Reader)Fix the issue of BE core dump caused by improper configuration of column_seperator and line_delimiter. (#33693)

2024-04-20 20:06:48 +08:00

test_csv_split_line1.csv

[fix](csv-reader)fix bug of multi-char delimiter in csv reader

2023-08-23 15:19:13 +08:00

test_csv_split_line2.csv

[fix](csv_reader)Fixed bug when parsing multi-character delimiters. (#24572)

2023-09-20 12:41:35 +08:00

test_csv_split_line3.csv

[fix](csv_reader)Fixed bug when parsing multi-character delimiters. (#24572)

2023-09-20 12:41:35 +08:00

test_csv_split_line4.csv

[fix](csv_reader)Fixed bug when parsing multi-character delimiters. (#24572)

2023-09-20 12:41:35 +08:00

test_csv_split_line.out

[fix](csv_reader)Fixed bug when parsing multi-character delimiters. (#24572)

2023-09-20 12:41:35 +08:00

test_csv_with_double_quotes.out

[fix](csv_reader) fix trim_double_quotes behavior change (#27882)

2023-12-03 22:57:55 +08:00

test_csv_with_enclose_and_escape.out

[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)

2023-08-15 09:23:53 +08:00

test_csv_with_skip_lines.out

…

test_decimal.csv

[Fix](mutli-catalog) Use decimal v3 type to fix decimal loss issue in multi-catalog module. (#18835)

2023-04-20 11:02:53 +08:00

test_decimal.json

[Fix](mutli-catalog) Use decimal v3 type to fix decimal loss issue in multi-catalog module. (#18835)

2023-04-20 11:02:53 +08:00

test_decimal.orc

[Fix](mutli-catalog) Use decimal v3 type to fix decimal loss issue in multi-catalog module. (#18835)

2023-04-20 11:02:53 +08:00

test_decimal.parquet

[Fix](mutli-catalog) Use decimal v3 type to fix decimal loss issue in multi-catalog module. (#18835)

2023-04-20 11:02:53 +08:00

test_default_value.csv

[regression-test](load) add streamload default value test (#21536)

2023-07-06 10:14:13 +08:00

test_double_quotes_with_enclose.csv

[fix](csv_reader) fix trim_double_quotes behavior change (#27882)

2023-12-03 22:57:55 +08:00

test_duplicate_json_keys.json

[Fix](JsonReader) fix json with duplicate key entry may result out of bound exception (#38147)

2024-07-19 22:53:02 +08:00

test_empty_file.csv

[test](streamload) add load empty file regression test (#34110)

2024-04-26 07:42:09 +08:00

test_error_url_1.csv

[branch-2.1](load) fix no error url if no partition can be found (#36831) (#37401)

2024-07-08 10:41:33 +08:00

test_error_url.csv

[fix](pipeline-load) fix no error url when data quality error and total rows is negative (#34072) (#34204)

2024-04-27 18:19:08 +08:00

test_group_commit_and_wal_back_pressure.csv.gz

[Fix](core) Fix wal space back pressure core and add regression test (#27311)

2023-11-27 15:10:26 +08:00

test_group_commit_and_wal_back_pressure.out

[Fix](core) Fix wal space back pressure core and add regression test (#27311)

2023-11-27 15:10:26 +08:00

test_group_commit_stream_load.out

[feature](insert) Support group commit stream load (#24304)

2023-09-26 20:57:02 +08:00

test_group_commit_wal_limit.csv.gz

[Enhancement](group commit) Add regression case for wal limit (#27949)

2023-12-06 14:23:50 +08:00

test_hdfs_json_load.out

[fix](hdfs)read 'fs.defaultFS' from core-site.xml for hdfs load which has no default fs (#34217) (#34372)

2024-05-01 00:31:49 +08:00

test_immutable_partition.csv

[Feature](load) Support mutable property for partition (#16036)

2023-02-18 23:09:34 +08:00

test_include_where_expr.json

[Fix](Plan)StreamLoad cannot be parsed correctly when it contains complex where conditions (#23874)

2023-09-05 11:26:59 +08:00

test_input_long_than_schema.csv

[fix](load) fix loaded rows error (#28061)

2023-12-07 14:43:51 +08:00

test_json_error.json

[improve](json_reader) add prompt when all fields is null (#27630)

2023-11-29 18:26:42 +08:00

test_json_load_default_value.out

[fix](load)Fix wrong default value for char and varchar of reading json data (#22626)

2023-08-05 12:47:27 +08:00

test_json_load.out

[Fix](JsonReader) fix json with duplicate key entry may result out of bound exception (#38147)

2024-07-19 22:53:02 +08:00

test_json_object_array.csv

[pick](simdjson) fix simdjson with object array when jsonroot is not empty (#38633)

2024-08-01 11:04:54 +08:00

test_keyword.csv

[fix](stream_load) fix stream load may failed caused by column name with keyword (#35822) (#37890)

2024-07-16 20:20:34 +08:00

test_line_delimiter.csv

…

test_load_block_to_single_tablet.csv

[bug](random distribution) fix data loss and incorrect in random distribution table #33962

2024-04-24 17:13:50 +08:00

test_load_block_to_single_tablet.out

[bug](random distribution) fix data loss and incorrect in random distribution table #33962

2024-04-24 17:13:50 +08:00

test_load_to_single_tablet.json

[opt](load)change load_to_single_tablet tablet search algorithm from random to round-robin (#25256)

2023-10-16 16:43:25 +08:00

test_load_with_decimal.out

[Fix](mutli-catalog) Use decimal v3 type to fix decimal loss issue in multi-catalog module. (#18835)

2023-04-20 11:02:53 +08:00

test_load_with_jsonpath.json

…

test_load_with_map_nested_array.out

[FIX](regresstest) fix test_map_nested_array csv file for id(#27105)

2023-11-17 04:20:02 -06:00

test_malformed_json_with_path.json

[Bug](json reader) object should stop processing when encounter error (#31159)

2024-02-21 13:53:32 +08:00

test_map_load_and_compaction.out

[Improve](complex-type) Support Count(complexType) (#17868)

2023-03-30 15:43:32 +08:00

test_map_load_and_function.out

[Fix](Serde) Fix content displayed by complex types in MySQL Client (#25946)

2023-11-01 23:48:55 +08:00

test_map_nested_array.csv

[FIX](regresstest) fix test_map_nested_array csv file for id(#27105)

2023-11-17 04:20:02 -06:00

test_map.csv

[Feature](map-type) Support stream load and fix some bugs for map type (#16776)

2023-02-19 15:11:54 +08:00

test_orc_case.orc

[Fix](load)Use lower case for load column names. #16422

2023-02-07 09:18:37 +08:00

test_parquet_case.parquet

[Fix](load)Use lower case for load column names. #16422

2023-02-07 09:18:37 +08:00

test_parquet_orc_case.out

[Conf](decimalv3) enable decimalv3 by default

2023-05-29 15:38:31 +08:00

test_pipeline_load.out

[case](regression) Test enable pipeline load (#28172)

2023-12-28 10:49:19 +08:00

test_publish_timeout.out

[case](regression) streamload publish timeout (#29457)

2024-01-07 19:50:16 +08:00

test_single_quote.csv

[fix](csv_reader) fix trim_double_quotes behavior change (#27882)

2023-12-03 22:57:55 +08:00

test_special_key_json.json

[regression-test](load) add something like $.tag.[a.b] key's json case (#35134)

2024-05-31 22:45:09 +08:00

test_stream_load1.csv

[feature](insert) Support group commit stream load (#24304)

2023-09-26 20:57:02 +08:00

test_stream_load2.csv

[feature](insert) Support group commit stream load (#24304)

2023-09-26 20:57:02 +08:00

test_stream_load3.csv

[feature](insert) Support group commit stream load (#24304)

2023-09-26 20:57:02 +08:00

test_stream_load_cast.out

[test](cast) add test for stream load cast (#33189)

2024-04-10 15:26:09 +08:00

test_stream_load_data_model.csv

[Enhancement](Load) Stream Load using SQL (#22509)

2023-08-08 13:49:04 +08:00

test_stream_load_empty_file.out

[test](streamload) add load empty file regression test (#34110)

2024-04-26 07:42:09 +08:00

test_stream_load_err_log_limit.csv

[Enhancement](load) Limit the number of incorrect data drops and add documents (#27727)

2023-12-22 10:43:18 +08:00

test_stream_load_err_log_limit.out

[Enhancement](load) Limit the number of incorrect data drops and add documents (#27727)

2023-12-22 10:43:18 +08:00

test_stream_load_hll_type.csv

[improvement](stream load)(cherry-pick) support hll_from_base64 for stream load column mapping (#36819)

2024-06-26 20:12:40 +08:00

test_stream_load_keyword.out

[fix](stream_load) fix stream load may failed caused by column name with keyword (#35822) (#37890)

2024-07-16 20:20:34 +08:00

test_stream_load_move_memtable.out

[feature](serde) support presto compatible output format (#37039) (#37253)

2024-07-04 13:56:05 +08:00

test_stream_load_multiple_times.csv

[Enhancement](Load) Stream Load using SQL (#22509)

2023-08-08 13:49:04 +08:00

test_stream_load_new_change_column_order.csv

[Enhancement](Load) Stream Load using SQL (#22509)

2023-08-08 13:49:04 +08:00

test_stream_load_new_column_number_mismatch.csv

[Enhancement](Load) Stream Load using SQL (#22509)

2023-08-08 13:49:04 +08:00

test_stream_load_new_column_separator.csv

[Enhancement](Load) Stream Load using SQL (#22509)

2023-08-08 13:49:04 +08:00

test_stream_load_new_column_type.csv

[Enhancement](Load) Stream Load using SQL (#22509)

2023-08-08 13:49:04 +08:00

test_stream_load_new_current_timestamp.csv

[Enhancement](Load) Stream Load using SQL (#22509)

2023-08-08 13:49:04 +08:00

test_stream_load_new_default_value.csv

[Enhancement](Load) Stream Load using SQL (#22509)

2023-08-08 13:49:04 +08:00

test_stream_load_new_function.csv

[Enhancement](Load) Stream Load using SQL (#22509)

2023-08-08 13:49:04 +08:00

test_stream_load_new_line_delimiter.csv

[Enhancement](Load) Stream Load using SQL (#22509)

2023-08-08 13:49:04 +08:00

test_stream_load_new_move_memtable.out

[feature](move-memtable)[7/7] add regression tests (#23515)

2023-08-26 17:52:10 +08:00

test_stream_load_new.out

[improvement](stream load)(cherry-pick) support hll_from_base64 for stream load column mapping (#36819)

2024-06-26 20:12:40 +08:00

test_stream_load_properties.out

[fix](nereids)change the decimal's precision and scale for cast(xx as decimal) (#36540)

2024-06-20 17:46:11 +08:00

test_stream_load.out

[feature](serde) support presto compatible output format (#37039) (#37253)

2024-07-04 13:56:05 +08:00

test_strict_mode_fail.csv

…

test_strict_mode.csv

…

test_time.data

…

test_two_phase_commit.csv

[Feature](StreamLoad2PC) Support commit and abort streamload2PC by label (#24613)

2023-09-25 22:21:27 +08:00

test_txt_special_delimiter.csv

[feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load (#22539)

2023-08-15 09:23:53 +08:00

test_txt_special_delimiter.out

…

trim_double_quotes.csv

[regression-test](TRIM_DOUBLE_QUOTES) add case for TRIM_DOUBLE_QUOTES (#26998)

2023-11-14 23:52:40 +08:00

unique_key_with_delete.csv

…

unique_key.csv.lz4

…

with_jsonb.json

[Improve](simdjson) put unescaped string value after parsed (#21866)

2023-07-20 10:33:17 +08:00

with_jsonpath.json

[Fix](json reader) fix rapidjson array->PushBack may take ownership… (#21988)

2023-07-21 17:02:01 +08:00