Commit Graph

9259 Commits

Author SHA1 Message Date
2b31fc1472 [fix](regression) segcompaction timeout too short (#16731) (#17565)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2023-03-13 11:19:21 +08:00
b9fac82fb1 [fix](regression) adjust regression pipeline config(tablet_create_timeout_second) for avoiding create partition timeout (#17668)
This pull request for bellow problem :
regression pipleline fail case always meet error "Failed to create partition. Timeout. Unfinished mark: 10003=57059", so adjust tablet_create_timeout_second to 100
2023-03-13 11:18:03 +08:00
5fccbac81b [fix](demo)add Sync full database for versions below doris 1.2 (#17669) 2023-03-13 11:17:29 +08:00
Pxl
16fc3a0e22 [Chore](compile) remove some unused static on inline function to reduce compile time (#17603)
remove some unused static on inline function to reduce compile time
2023-03-13 11:11:59 +08:00
782001c75b [fix](planner) project should be done inside subquery (#17630)
WITH t0 AS(
SELECT report.date1 AS date2 FROM(
SELECT DATE_FORMAT(date, '%Y%m%d') AS date1 FROM cir_1756_t1
) report GROUP BY report.date1
),
t3 AS(
SELECT date_format(date, '%Y%m%d') AS date3
FROM cir_1756_t2
)
SELECT row_number() OVER(ORDER BY date2)
FROM(
SELECT t0.date2 FROM t0 LEFT JOIN t3 ON t0.date2 = t3.date3
) tx;

The DATE_FORMAT(date, '%Y%m%d') was calculated in GROUP BY node, which is wrong. This expr should be calculated inside the subquery.
2023-03-13 11:10:27 +08:00
55c42da511 [Feature](array) Support array<decimalv3> data type (#16640) 2023-03-13 10:48:13 +08:00
3a6c0e7867 [fix](regression) fix test_array_export and test_map_export dir conflict #17636
regression test test_array_export and test_map_export use same output dir, if they run at the same time, the cases will failed.
2023-03-13 10:35:50 +08:00
39b5682d59 [Pipeline](shared_scan_opt) Support shared scan opt in pipeline exec engine 2023-03-13 10:33:57 +08:00
edb2d90852 [fix](routine load) fix ROUTINE LOAD bug,kafka commit a lack of one(#17282) (#17291)
Co-authored-by: hugoluo <hugoluo@tencent.com>
2023-03-13 10:20:59 +08:00
a0a2809324 [Enhancement](multi-catalog) support hms event deserialization for HDP/CDH Hive versions. (#17660)
Some HDP/CDH Hive versions use gzip to compress the message body of hms NotificationEvent,
so com.qihoo.finance.hms.event.MetastoreEventFactory can not transfer it rightly.
2023-03-13 09:47:28 +08:00
93a865c3e8 [improvement](join) Avoid reading from left child while hash table is empty(right join) (#17655)
When the right (build) side is empty in a right outer join, there is no need to read data from the left child.
2023-03-13 09:03:17 +08:00
47cfc81925 [fix docs] (#17634)
Co-authored-by: shenshoucheng <shenshoucheng@jd.com>
2023-03-13 08:06:33 +08:00
33059d92cc [docs](doc) fix faq docs (#17707) 2023-03-13 08:05:12 +08:00
7948c8ede2 [bug](function) Fix bug in the process of generating template functions #17676
If users compile fe with python version <= 2.6

They will meet a compile error in gen_builtins_functions.py with following msg:
2023-03-13 07:29:32 +08:00
6386458498 [Refactor](exec) remove unless attr of slot ref (#17688)
Remove unless attr of slot ref
2023-03-12 23:45:32 +08:00
b0d1166989 [fix](meta) fix concurrent modification exception and potential NPE (#17602) 2023-03-12 22:12:07 +08:00
46dcf69644 [fix](jdbc-catalog) avoid calculate driver's md5 when replaying edit log (#17693) 2023-03-12 22:11:45 +08:00
54e5c71e52 [fix](planner) Fix NPE when update stats by profile 2023-03-12 21:40:47 +08:00
a651926ba9 [fix](fqdn) Add UnknownHostException handle logic in FQDNManager to avoid that active ip could be incorrectly assigned to dead be or dead fe (#17689)
1.if be is dead and be ip not changed by FQDNManager,A situation may occur that after a while the old ip is used by other new alive pod,this may cause two be share same ip which is unexpected.
2.when enable_fqdn is false, user can still set hostname in be when add backend

Co-authored-by: caiconghui1 <caiconghui1@jd.com>
2023-03-12 21:12:33 +08:00
13e05c4a5d [Enhencement](stream load) add some regression test for json format streamload (#17520) 2023-03-12 20:13:07 +08:00
0d05e4cce0 [Improvement](multi-catalog) The interface of external Splitter. WIP (#17390)
This is PR introduce splitter interface external table.
The splitter interface contain one method getSplits, which is used by QueryScanProvider to get the external file split. 
For Hive/Iceberg/TVF, a split is a file block. For ES, it is a shard.
This PR also move the getSplits logic in FileScanProviderIf to the new Splitter interface.
In the future, we may unify internal table as well.
2023-03-12 20:11:08 +08:00
455c800405 [feature](parquet-reader) add rle bool and delta decoder to read AWS Glue (#17112)
Support delta encoding and rle(bool) to read Glue data
add delta bit pack decoder,
add delta length byte array decoder,
add delta byte array decoder.
add rle bool decoder.

We find some data type is read with delta encoding on AWS Glue, so it should be supported.
The definition of delta encoding can refer to the delta encoding in parquet.
2023-03-12 20:09:58 +08:00
a452db35da [improvement](filecache)Change the hash field of the backend (#17499)
ip of backend may change
use id as a hash field
2023-03-12 20:04:25 +08:00
b93e553958 [enhance](Nereids): allow empty hash condition (#17699) 2023-03-12 18:51:22 +08:00
11fbe07221 [refactor](Nereids) Refactor all rewrite logical unit tests by match-pattern (#17691) 2023-03-12 18:49:12 +08:00
9b687026bd [Doc](TLS) add doc for TLS connection (#17683) 2023-03-12 10:01:07 +08:00
d774162a53 [minor](Nereids): rename rule (#17509) 2023-03-12 00:17:07 +08:00
4566de1cec Enhancement in some Hyperlinks (#17654) 2023-03-11 22:56:43 +08:00
9745ee60a7 [fix](priv) fix bug of grant priv on ctl.db.* not work (#17612)
currently, when use grant xxx_priv on ctl.db.* to user_a, it does not work. When user_a switch to ctl,
he cannot see or use any database.
2023-03-11 22:27:26 +08:00
692d510edb [fix](schema_hash) remove useless schema_hash param in tablet and replica url (#17489)
Co-authored-by: caiconghui1 <caiconghui1@jd.com>
2023-03-11 21:34:47 +08:00
d7cb5cf3db [feature](nereids) add session var: dump_nereids_memo (#17666)
* dump_nereids_memo

* print groupexpr id
2023-03-11 13:40:15 +08:00
3231fab8c2 [feature](nereids) add unique id for groupExpression and plan node (#17628)
* add unqiue id for groupExpression and plan node

* fix ut
2023-03-11 13:23:41 +08:00
db9692a114 [feature](Nereids): convert CrossJoin to InnerJoin. (#17681) 2023-03-11 13:23:28 +08:00
9c7854f1ff [Enhancement](k8s) Add k8s yaml demo (#17281) 2023-03-11 10:56:57 +08:00
Pxl
8328ab69ad [Chore](Materialized-View) add some mv regression test case (#17345)
1. add some mv regression test case
2. rename materialized_view_p0 to mv_p0 (avoid create database failed because long db name)
2023-03-11 10:55:11 +08:00
697cba9a85 [fix](broker-load) fix broker's Dockerfile (#17657)
there is some spelling mistake in broker's Dockerfile and need to fix it.
2023-03-11 10:43:09 +08:00
a74ef2377f typo fix in kyuubi doc (#17672) 2023-03-11 09:11:10 +08:00
48a2fe68ad [typo](docs) Fix some display errors (#17663)
* [fix](docs) fix some errors in docs
2023-03-11 09:10:48 +08:00
3745e6c18a [fix](Nereids): order of project's logical properties is different with that of project expression (#17648) 2023-03-11 00:26:54 +08:00
051ab7a9c6 [refactor](Nereids): refactor Join-Dependent Predicate Duplication. (#17653) 2023-03-10 22:19:45 +08:00
566d133610 [enhancement](Nereids) Refactor EliminateLimitTest and EliminateFilterTest by match-pattern (#17631) 2023-03-10 21:24:36 +08:00
6dcd791b74 [feature](struct-type) support CAST AS Struct type (#17553)
1. add support `CAST AS Struct` from Struct type;
2. fix crash while `CAST('{}' AS Struct)`;
3. `CAST('' AS complext_type)` should return NULL instead of empty object;

---------

Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2023-03-10 21:21:16 +08:00
2739a44eaf [fix](segcompaction) heap overflow when doing segcompaction for cancelling load(#17529)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2023-03-10 20:52:05 +08:00
47e9217c1e [improvement](pipeline)Avoid duplicate trigger teamcity build (#17647)
* add clickbench and arm pipeline trigger;test merge check


* set compile required; add clickbench and arm pipeline trigger to buildall;

* avoid duplicate builds

* simplify auto trigger code and avoid repeated triggers 0310
2023-03-10 19:56:14 +08:00
948654ad38 [fix](ui)format the text file of profile #17645 2023-03-10 19:54:28 +08:00
9cfa61b402 [Enhancement](HttpServer) Provide authentication interface for BE (#17073)
Add an authentication interface in FE for BE
2023-03-10 16:34:47 +08:00
9ae5ec4dc5 [fix](nereids) PushdownExpressionsInHashCondition contains duplicate column and WindowExpression miss column stats (#17624)
tpcds: q47 and q57
1. PushdownExpressionsInHashCondition:project contains duplicate column
2. WindowExpression stats caclucate: miss column stats
2023-03-10 16:08:43 +08:00
365c8eed7e [fix](function) width_bucket should get min and max from each tuple (#17466) 2023-03-10 13:14:12 +08:00
739e043c8d [fix](publish) add retry publish when succeed replica num less than quorum and transaction not VISIBLE (#17453)
for some reasons, transaction pushlish succeed replica num less than quorum,
this transaction's status can not to be VISIBLE, and this publish task of this
replica of this tablet on this backend need retry publish success to
make transaction VISIBLE when last publish failed.
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2023-03-10 12:02:15 +08:00
a79b8ede88 [Bug](ColumnArray) Fix array column replicate replicate_offsets not matched (#17616)
the input replicate_offsets should be the same size as ColumnArray's offset.
```
IColumn::Offsets replicate_offsets(get_offsets().size(), 0);
// |---------------------|-------------------------|-------------------------|
// [0, begin)             [begin, begin + count_sz)  [begin + count_sz, size())
//  do not need to copy    copy counts[n] times       do not need to copy
```

we should
2023-03-10 11:52:22 +08:00