aa4dbbedc7
[pipelineX](bug) Fix dead lock in exchange sink operator ( #24947 )
2023-09-27 15:40:25 +08:00
87a30dc41d
[feature-wip](arrow-flight)(step3) Support authentication and user session ( #24772 )
2023-09-27 14:53:58 +08:00
26818de9c8
[feature](jni) support complex types in jni framework ( #24810 )
...
Support complex types in jni framework, and successfully run end-to-end on hudi.
### How to Use
Other scanners only need to implement three interfaces in `ColumnValue`:
```
// Get array elements and append into values
void unpackArray(List<ColumnValue> values);
// Get map key array&value array, and append into keys&values
void unpackMap(List<ColumnValue> keys, List<ColumnValue> values);
// Get the struct fields specified by `structFieldIndex`, and append into values
void unpackStruct(List<Integer> structFieldIndex, List<ColumnValue> values);
```
Developers can take `HudiColumnValue` as an example.
2023-09-27 14:47:41 +08:00
1b0e3246ea
[pipelineX](fix) Fix exception reporting and Nereids plan ( #24936 )
2023-09-27 13:15:40 +08:00
c04e5bac39
[bug](pipelineX) fix java-udaf failed with open pipelineX ( #24939 )
2023-09-27 13:14:10 +08:00
452318a9fc
[Enhancement](streamload) stream tvf support user specified label ( #24219 )
...
stream tvf support user specified label
example:
curl -v --location-trusted -u root: -H "sql: insert into test.t1 WITH LABEL label1 select c1,c2 from http_stream(\"format\" = \"CSV\", \"column_separator\" = \",\")" -T example.csv http://127.0.0.1:8030/api/_http_stream
return:
{
"TxnId": 2064,
"Label": "label1",
"Comment": "",
"TwoPhaseCommit": "false",
"Status": "Success",
"Message": "OK",
"NumberTotalRows": 2,
"NumberLoadedRows": 2,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 27,
"LoadTimeMs": 152,
"BeginTxnTimeMs": 0,
"StreamLoadPutTimeMs": 83,
"ReadDataTimeMs": 92,
"WriteDataTimeMs": 41,
"CommitAndPublishTimeMs": 24
}
2023-09-27 12:09:35 +08:00
24ee3607e1
[Bug](pipeline) nullprt may be close the sink if init failed ( #24926 )
2023-09-27 09:11:06 +08:00
a689a2fbb1
[pipelineX](fix) Fix projection expression ( #24923 )
2023-09-26 21:48:28 +08:00
55d1090137
[feature](insert) Support group commit stream load ( #24304 )
2023-09-26 20:57:02 +08:00
fe2879d8fe
[fix](merge-on-write) MergeIndexDeleteBitmapCalculator stack overflow ( #24913 )
2023-09-26 20:32:23 +08:00
77e864df12
[enhancement](delete) use column id in delete push task instead of column name ( #24549 )
2023-09-26 19:54:55 +08:00
ded8ba108f
[test](be-ut) skip some be ut case ( #24917 )
...
skip be ut "VTimestampFunctionsTest.convert_tz_test"
2023-09-26 19:51:15 +08:00
28869b0f82
[fix](Outfile) Use data_type_serde to export data to orc file format ( #24812 )
2023-09-26 19:46:42 +08:00
94082ae59c
[Fix](inverted index) fix tokenize function coredump ( #24896 )
2023-09-26 17:31:10 +08:00
156b7b80a4
[pipelineX](bug) Fix scan operator if aggregation type is COUNT_ON_INDEX ( #24909 )
2023-09-26 16:42:29 +08:00
1abda1c446
[Fix](merge-on-write) Correct the alignment process when the existing rows with same key has marked delete sign ( #24877 )
2023-09-26 16:09:20 +08:00
082bcd820b
[feature](insert) Support wal for group commit insert ( #23053 )
2023-09-26 14:46:24 +08:00
422170ccb1
[Bugfix](tablet) change WriteCooldownMetaExecutors's thread pool type to prevent singleton's dtor order ( #23834 )
2023-09-26 14:29:35 +08:00
a3427cb822
[pipelineX](fix) Fix nested loop join operator ( #24885 )
2023-09-26 13:27:34 +08:00
733b71828c
[fix](pipelineX) fix do not set per_fragment_instance_idx ( #24890 )
2023-09-26 13:10:30 +08:00
513e37bdbf
[pipelineX](node)support jdbc scan operator ( #24851 )
2023-09-26 10:02:51 +08:00
8ee2e40812
[enhancement](zone_map_index) use direct_copy_may_cut instead of direct_copy( #24880 )
2023-09-25 22:32:18 +08:00
8191cd1dad
[Bug](ScanNode) Fix potential incorrect query result caused by concurrent NewOlapScanNode initialization and Compaction ( #24638 )
...
* Optimize fetch delete predicates
* Fix incorrect query result when compaction eliminate delete predicates between `NewOlapScanNode::_init_scanners` and `NewOlapScanner::init`
* Fix be ut
2023-09-25 22:24:35 +08:00
8d4fd76a16
[Feature](StreamLoad2PC) Support commit and abort streamload2PC by label ( #24613 )
2023-09-25 22:21:27 +08:00
b38b8b4494
[pipelineX](fix) Fix BE crash caused by join and constant expr ( #24862 )
2023-09-25 21:01:09 +08:00
6502da8917
[bugfix](restore) add partition id into convert_rowset_ids() ( #24834 )
2023-09-25 20:07:24 +08:00
8679095e5c
[feature](debug) support debug point used in debug code ( #24502 )
2023-09-25 17:56:12 +08:00
3b4d8b4ac8
[pipelineX](feature) Support schema scan operator ( #24850 )
2023-09-25 14:42:25 +08:00
9412775686
remove useless variable in scanctx ( #24849 )
...
remove useless variable in scanctx
2023-09-25 14:36:18 +08:00
1b95ce1d93
[feature](json-function) add json_insert, json_replace, json_set functions ( #24384 )
...
[feature](json-function) add three json funcitons
2023-09-25 12:52:29 +08:00
c9ef5ef2b1
[refactor](profile) refactor join node profile when build side shared hash table ( #24785 )
...
refactor join node profile when build side shared hash table
2023-09-25 10:28:16 +08:00
39e6512a21
[bug](scanner) Fix memory out of bound in scanner scheduler ( #24840 )
2023-09-25 09:58:26 +08:00
9579634eac
[Debug](pipeline) add log of pipeline scan bug ( #24804 )
2023-09-25 08:38:31 +08:00
aea0279f25
[chore](build) Fix issues with GLIBC dependency ( #24837 )
...
Linking with libabsl_random_internal_seed_material.a instead of glibc to fix this issue #24838 .
2023-09-25 08:21:19 +08:00
900b06d14e
[improvement](disk balancer) reduce disk balance sensitivity ( #24637 )
2023-09-24 22:02:53 +08:00
fa0e0a606b
[Fix](autoinc) Ensure that _fetch_autoinc_id_executor is destructed after the destructions of AutoIncIDBuffers ( #24832 )
2023-09-24 21:44:02 +08:00
27eed937b3
[pipelineX](es scan) Support ES scan operator ( #24824 )
...
Support ES scan operator
2023-09-24 00:32:38 +08:00
5a4d51716c
[fix](pipelineX) fix null ptr when unionoperator only have constexpr #24822
2023-09-23 20:28:38 +08:00
11b6fb9d10
[PipelineX](feature) Support data gen operator ( #24823 )
2023-09-23 19:28:47 +08:00
a0de08255d
[pipelineX](profile) Add necessary metrics ( #24820 )
2023-09-23 19:27:45 +08:00
329c22da00
[pipelineX](feature) Support table function operator ( #24818 )
2023-09-23 19:26:53 +08:00
ce79711b0d
[FIX](serde) fix map/array deserialize string with quote pair ( #24808 )
2023-09-23 01:12:20 +08:00
3d786c58de
[pipelineX](sink) Support async writer ( #24793 )
2023-09-22 23:04:23 +08:00
3bf95b2cfe
[fix](merge-on-write) fix duplicate key in schema change ( #24782 )
2023-09-22 19:50:31 +08:00
f4b1e2b343
[Fix](core) Fix segment cache core when output rowset is nullptr ( #24778 )
2023-09-22 19:48:42 +08:00
ac55d45f79
[Fix](topn opt) fix heap use after free when shrink in fetch phase ( #24774 )
2023-09-22 19:48:05 +08:00
4007622ad9
[Improve](inverted index) improve match performance without index ( #24751 )
2023-09-22 18:45:11 +08:00
c346f4d638
[Improvement](Storage) Lazy init mini_download dir ( #24649 )
2023-09-22 16:23:26 +08:00
263506f8ab
[refactor](pipelineX) add MultiCast operator ( #24656 )
2023-09-22 15:41:14 +08:00
3c99743bf2
[enhancement](csv_reader)Optimize the reading efficiency of nullable (string) columns. ( #24698 )
...
Optimize the performance of stream load tsv by reducing virtual function calls .
(Optimize read performance of nullable (string) columns by reducing virtual function calls.)
before : 600+ s
after : 560+ s
2023-09-22 13:44:37 +08:00
