database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
13,810 Commits 1 Branch 0 Tags
bf808e9aa68c3f1761dfdcec8c9ffae8231ec0d5
Commit Graph

6112 Commits

Author SHA1 Message Date
jakevin
bf808e9aa6 [fix](Nereids): tolerate DateLike overflow in SQL CAST/CONVERT (#24943) 
- explicit type cast, we need tolerate overflow and convert it to be NULL
- implicit type cast, throw exception
 2023-09-28 12:11:50 +08:00
Siyang Tang
188d9ab94e [enhancement](statistics) collect table level loaded rows on BE to make RPC light weight (#24609) 2023-09-28 10:51:50 +08:00
minghong
42207df89f [refactor](nereids)update NormalizeRepeat comments (#24893) 
update NormalizeRepeat comments
 2023-09-28 10:42:16 +08:00
starocean999
584646c054 [improvement](nereids)dphyper GraphSimplifier should consider missed edges when estimating join cost (#21747) 2023-09-28 09:30:57 +08:00
airborne12
732f821c15 [Fix](inverted index) make parser mode coarse grained by default (#24949) 2023-09-27 21:04:41 +08:00
Liqf
d4e823950a [bug](json)Fix some problems of json function on Nereids (#24898) 
Fix some problems of json_length and json_contains function on Nereids
fix wrong result of json_contains function
Regression test jsonb_p0 to enable Nereids
 2023-09-27 21:01:45 +08:00
meiyi
391a4e29eb [fix](schema) Table column order is changed if add a column and do truncate (#24981) 2023-09-27 20:59:11 +08:00
morrySnow
63b283a848 [fix](Nereids) init Date/DateV2Literal should check non-zero time fields (#24971) 2023-09-27 20:48:36 +08:00
Gabriel
1fb9022d07 [pipelineX](bug) Fix meta scan operator (#24963) 2023-09-27 20:34:47 +08:00
xzj7019
bb7f8d18a8 [fix](nereids) push down filter through partition topn (#24944) 
support pushing down filter through partition topn if the filter can pass through window.
fix CreatePartitionTopNFromWindow bug which may generate two partition topn unexpectly.
case:
select * from (select c2, row_number() over (partition by c2) as rn from t1) T where rn<=1 and c2 = 1;
before this pr:
| PhysicalResultSink                       |
| --PhysicalDistribute                     |
| ----filter((rn <= 1))                    |
| ------PhysicalWindow                     |
| --------PhysicalQuickSort                |
| ----------PhysicalDistribute             |
| ------------PhysicalPartitionTopN        |
| --------------filter((T.c2 = 1))         |
| ----------------PhysicalPartitionTopN    |
| ------------------PhysicalProject        |
| --------------------PhysicalOlapScan[t1] |
+------------------------------------------+
after:

| PhysicalResultSink                     |
| --PhysicalDistribute                   |
| ----filter((rn <= 1))                  |
| ------PhysicalWindow                   |
| --------PhysicalQuickSort              |
| ----------PhysicalDistribute           |
| ------------PhysicalPartitionTopN      |
| --------------PhysicalProject          |
| ----------------filter((T.c2 = 1))     |
| ------------------PhysicalOlapScan[t1] |
+----------------------------------------+
 2023-09-27 19:38:04 +08:00
morrySnow
00786a3295 [fix](Nereids) could not prune datev1 partition column (#24959) 
because storage engine could not process date comparison predicates.
we convert it to datetime comparison predicates.
however, partition prunner could not process cast(slot) cp literal.
so, we convert back in partition pruner to let it work well.

TODO:
move convert date to datetime in translate stage
and only convert predicates for storage engine.
 2023-09-27 18:41:56 +08:00
Mryange
5d138b6928 [remove](function) make execute_impl const and remove running_difference function (#24935) 2023-09-27 18:17:28 +08:00
zy-kkk
100d76510c [Fix](HttpServer) Refactor API Endpoints to Only Allow GET Requests for Enhanced Security (#24855) 2023-09-27 17:10:11 +08:00
LiBinfeng
00e8d1c3b4 [Fix](Planner) disable bitmap type in compare expression (#24792) 
Problem:
be core because of bitmap calculation.

Reason:
when be check failed, it would core directly.

Example:
SELECT id_bitmap FROM test_bitmap WHERE id_bitmap IN (NULL) LIMIT 20;

Solved:
Forbidden this kind of expression in fe when analyze. And also forbid bitmap type comparing in other unsupported expressions.
 2023-09-27 16:57:06 +08:00
bigben0204
0227292c85 [bug](profile) query profile api of fe cann't get result if non-root user query on the other fe #24858 (#24914) 
Issue Number: #24858

If isAllNode is true, the api should only distribute the query to all fe and do not run checkAuthByUserAndQueryId.
If isAllNode is false, the api queries profile on the fe, at this time the api should run checkAuthByUserAndQueryId.
 2023-09-27 16:50:41 +08:00
谢健
9562e280af [enhancement](Nereids): remove stats derivation in CostAndEnforce job (#24945) 
1. remove stats derivation in CostAndEnforce job
2. enforce valid for each stats after estimating
 2023-09-27 16:31:03 +08:00
Xinyi Zou
87a30dc41d [feature-wip](arrow-flight)(step3) Support authentication and user session (#24772) 2023-09-27 14:53:58 +08:00
Ashin Gau
26818de9c8 [feature](jni) support complex types in jni framework (#24810) 
Support complex types in jni framework, and successfully run end-to-end on hudi.
### How to Use
Other scanners only need to implement three interfaces in `ColumnValue`:
```
// Get array elements and append into values
void unpackArray(List<ColumnValue> values);

// Get map key array&value array, and append into keys&values
void unpackMap(List<ColumnValue> keys, List<ColumnValue> values);

// Get the struct fields specified by `structFieldIndex`, and append into values
void unpackStruct(List<Integer> structFieldIndex, List<ColumnValue> values);
```
Developers can take `HudiColumnValue` as an example.
 2023-09-27 14:47:41 +08:00
xzj7019
a1ab8f96a1 [fix](nereids) mark two phase partition topn global to notice be passthrough logic (#24886) 
mark partition topn phase to notice be to handle passthrough logic well, this pr is fe part code.
be side logic: the the phase equals to PTopNPhase.TWO_PAHSE_GLOBAL, it should skip the bypass logic and do the second phase ptopn operation anyway.
 2023-09-27 14:08:59 +08:00
Gabriel
1b0e3246ea [pipelineX](fix) Fix exception reporting and Nereids plan (#24936) 2023-09-27 13:15:40 +08:00
zzzzzzzs
452318a9fc [Enhancement](streamload) stream tvf support user specified label (#24219) 
stream tvf support user specified label
example:

curl -v --location-trusted -u root: -H "sql: insert into test.t1 WITH LABEL label1 select c1,c2 from http_stream(\"format\" = \"CSV\", \"column_separator\" = \",\")" -T example.csv http://127.0.0.1:8030/api/_http_stream
return:

{
    "TxnId": 2064,
    "Label": "label1",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 2,
    "NumberLoadedRows": 2,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 27,
    "LoadTimeMs": 152,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 83,
    "ReadDataTimeMs": 92,
    "WriteDataTimeMs": 41,
    "CommitAndPublishTimeMs": 24
}
 2023-09-27 12:09:35 +08:00
Pxl
18b5f70a7c [Bug](materialized-view) enable rewrite on select materialized index with aggregate mode (#24691) 
enable rewrite on select materialized index with aggregate mode
 2023-09-27 11:30:36 +08:00
minghong
a8f312794e [feature](nereids)support stats estimation for is-null predicate (#24764) 
1. condition order: filter/hashCondition/otherCondition,
2. update regression out
3. remove tpch_sf500 shape case(covered by tpch sf1000)
4. implement is-null stats estimation
5. update ssb shape
 2023-09-27 10:04:35 +08:00
starocean999
b9928a017b [fix](fe)add variant type in decimalv3's compatibilityMatrix table (#24749) 2023-09-27 10:03:13 +08:00
starocean999
0dd57b1982 [feature](nereids)support subquey in join condition (#24598) 2023-09-27 10:02:56 +08:00
zzzzzzzs
6d27a016b9 [Improvement](regression-test) add http_stream case (#24930) 2023-09-27 09:55:52 +08:00
zzzzzzzs
2e8f9a6429 [improvement](streamload) QeProcessorImpl#registerQuery should be removed (#24836) 2023-09-27 09:48:27 +08:00
DongLiang-0
f7f359d36c [fix](analysis)fix use regex determine whether time part exists may cause backtracking (#24882) 2023-09-27 09:26:42 +08:00
Jack Drogon
f59e69b852 [feature](autobucekt) Add support autobucket min buckets config Add support autobucket (#24920) 
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
 2023-09-27 08:13:53 +08:00
Guangdong Liu
504ab2d726 [fix](es-catalog) If the returned data is incorrect, it will be directly unified as the index was not found (#24831) 2023-09-26 22:59:38 +08:00
jakevin
841e6188bf [fix](Nereids): fix roundFloor (#24901) 
"roundFloor()" does not consider the issue of carry.
 2023-09-26 22:37:42 +08:00
morrySnow
90c5461ad2 [fix](Nereids) let dml work well (#24748) 
Co-authored-by: sohardforaname <organic_chemistry@foxmail.com>

TODO:
1. support agg_state type
2. support implicit cast literal exception
3. use nereids execute dml for these regression cases:

- test_agg_state_nereids (for TODO 1)
- test_array_insert_overflow (for TODO 2)
- nereids_p0/json_p0/test_json_load_and_function (for TODO 2)
- nereids_p0/json_p0/test_json_unique_load_and_function (for TODO 2)
- nereids_p0/jsonb_p0/test_jsonb_load_and_function (for TODO 2)
- nereids_p0/jsonb_p0/test_jsonb_unique_load_and_function (for TODO 2)
- json_p0/test_json_load_and_function (for TODO 2)
- json_p0/test_json_unique_load_and_function (for TODO 2)
- jsonb_p0/test_jsonb_load_and_function (for TODO 2)
- jsonb_p0/test_jsonb_unique_load_and_function (for TODO 2)
- test_multi_partition_key (for TODO 2)
 2023-09-26 21:08:24 +08:00
zzzzzzzs
a6a0e78f32 [Enhancement](streamload) stream tvf support compress (#24303) 2023-09-26 20:58:20 +08:00
meiyi
55d1090137 [feature](insert) Support group commit stream load (#24304) 2023-09-26 20:57:02 +08:00
yujun
e9eaf49db5 [feature](alter colocate group) Support alter colocate group replica allocation (#23320) 2023-09-26 19:55:25 +08:00
Siyang Tang
77e864df12 [enhancement](delete) use column id in delete push task instead of column name (#24549) 2023-09-26 19:54:55 +08:00
starocean999
04bf9bce54 [fix](planner)update explode slot's nullable info in analyze phase (#24879) 2023-09-26 18:14:04 +08:00
huanghaibin
082bcd820b [feature](insert) Support wal for group commit insert (#23053) 2023-09-26 14:46:24 +08:00
jakevin
983f948ec4 [fix](Nereids): fix SimplifyComparisonPredicate (#24899) 
#16061 exist BUG, it causes wrong cast
 2023-09-26 14:28:03 +08:00
yagagagaga
43433c3fab [improvement](processlist) Add more information for show processlist (#24000) 2023-09-26 14:25:42 +08:00
jakevin
6890e5bf0f [test](Nereids): add test for SimplifyComparisonPredicate (#24865) 2023-09-26 12:50:31 +08:00
morrySnow
eb9d68a640 [opt](Nereids) add json literal for constant folding (#24878) 2023-09-26 11:37:53 +08:00
lsy3993
1f8e0b48bc [fix](S3)delete main function because hardcoded ip is not safe (#24872) 2023-09-26 10:49:16 +08:00
zy-kkk
33630e1c4d [fix](multi-catalog) SecureRandom is used to replace Random in FederationBackendPolicy to enhance security (#24860) 2023-09-26 10:44:22 +08:00
zy-kkk
e887653dde [fix](auth) Remove printing passwords in the log (#24866) 2023-09-26 10:17:44 +08:00
jakevin
e4c0c98efa [fix](Nereids): round microsecond when specify scale of microsecond (#24854) 2023-09-26 10:11:53 +08:00
Euporia
b9472bf5d3 [fix](LOG)change printStackTracker to LOG (#24884) 2023-09-26 08:28:53 +08:00
yujun
60ce1ed8a3 [improvement](tablet clone) fix balanced new replica will be removed when load txn continuously and none-stop (#24845) 2023-09-25 22:51:08 +08:00
HHoflittlefish777
8d4fd76a16 [Feature](StreamLoad2PC) Support commit and abort streamload2PC by label (#24613) 2023-09-25 22:21:27 +08:00
Guangdong Liu
158570787a [fix](multi-catalog)The external catalog does not have verification parameters, causing an error to be reported when creating the catalog. (#24393) 2023-09-25 21:35:52 +08:00