Commit Graph

2686 Commits

Author SHA1 Message Date
3e686f4306 [feature](javaudf)support no input java udf (#24457)
* support no input java udf

* add license
2023-09-25 18:56:44 +08:00
90791f0b19 [fix](nereids)fix bug of exists subquery with limit clause (#24630)
create table t1(c1 int, c2 int);
create table t2(c1 int, c2 int);
insert into t1 values (1,1);
insert into t2 values (1,1);

select * from t1 where exists (select * from t2 where t1.c1 = t2.c1 limit 0);

the result should be empty set.
2023-09-25 17:15:08 +08:00
xfz
1b95ce1d93 [feature](json-function) add json_insert, json_replace, json_set functions (#24384)
[feature](json-function) add three json funcitons
2023-09-25 12:52:29 +08:00
cdc8189697 [Impro](regression-test) More test cases for function round (#24791) 2023-09-25 10:39:45 +08:00
ef72321878 [fix](regression-text)fix test_path_partition_keys regression test (#24796)
fix test_path_partition_keys regression test
2023-09-24 23:32:33 +08:00
b9496b2a8f [feature](docker) regression test support run suite in docker (#24508) 2023-09-24 21:44:16 +08:00
27eed937b3 [pipelineX](es scan) Support ES scan operator (#24824)
Support ES scan operator
2023-09-24 00:32:38 +08:00
9cd9e195d8 [test](load) add some s3 load regression test (#24399) 2023-09-23 22:40:45 +08:00
ce79711b0d [FIX](serde) fix map/array deserialize string with quote pair (#24808) 2023-09-23 01:12:20 +08:00
923402df71 [test](fix) fix case bug for lateral_view.groovy (#24779) 2023-09-22 21:21:34 +08:00
f3062e3d74 [fix](insert) Make insert group commit regression stable (#24584) 2023-09-22 20:18:14 +08:00
ac55d45f79 [Fix](topn opt) fix heap use after free when shrink in fetch phase (#24774) 2023-09-22 19:48:05 +08:00
35c0697a60 [fix](planner)SelectStmt's constructor should initialize originSelectList member (#24755)
resetSelectList method will use originSelectList to recover the origin select list. If the originSelectList is lost in constructor, the resetSelectList will fail to recover and make the analyze process fail.
2023-09-22 19:31:29 +08:00
eb9d334b55 [opt](nereids) enable runtime filter prune by default (#24717)
add stats back to physical plan in post processors. stats are used by rf-prune
set enable_runtime_filter_prune true by default
2023-09-22 19:11:10 +08:00
263506f8ab [refactor](pipelineX) add MultiCast operator (#24656) 2023-09-22 15:41:14 +08:00
74bba4bdaf [enhancement](regression-test) Add routine load case (#24536) 2023-09-22 14:55:01 +08:00
4edba083c0 [feat](Nereids) Support tablesample syntax (#23717)
Add table sample support like such SQL:

```sql
select * from test_table_sample_tbl tablesample(4 rows);
select * from test_table_sample_tbl t tablesample(20 percent);
select * from test_table_sample_tbl t tablesample(20 percent) repeatable 2;
```

This function has already been implemented in legacy planner, more detailed description could be found here: https://github.com/apache/doris/pull/10170
2023-09-22 14:24:07 +08:00
320fc1481a [fix](Nereids) some expression not cast in in predicate (#24680)
1. should use castIfNotSameType in InPredicate and CaseWhen
2. StringLikeLiteral should override equals to ignore type
2023-09-22 12:58:33 +08:00
034582bb64 [pipelineX](fix) Fix broadcast dependency hanging (#24740) 2023-09-22 12:24:32 +08:00
22616d125d [function](bitmap) add function alias bitmap_andnot and bitmap_andnot_count (#24771) 2023-09-22 12:18:31 +08:00
b5f6ace204 [fix](planner) ctas should not change any meta of column in source table (#24767)
if previous PR #22770. we try to fix incorrect nullable in target table.
However we changed nullable info of column in source table unexpectly
2023-09-22 11:49:38 +08:00
6f961ba0e9 [Enhance](external)add prepare hive data in case (#24703) 2023-09-22 11:19:46 +08:00
090be20ca4 [cases](regresstests) add negative case for agg table and fix agg table support replace typ… #24715
add negative case for agg table
fix agg table support replace agg type for complex type , and Now We only support complex type with agg state for replace only
fix test output
2023-09-22 09:05:20 +08:00
a17034b7ff [fix](planner) do not support UDF without paramter (#24730)
for example:
CREATE ALIAS FUNCTION f() WITH PARAMETERS() AS now();
2023-09-21 22:23:21 +08:00
58ab25ccaa Revert "[Feature](merge-on-write)Support ignore mode for merge-on-write unique table (#21773)" (#24731)
This reverts commit 3ee89aea35726197cb7e94bb4f2c36bc9d50da84.
2023-09-21 21:01:28 +08:00
a48b19ceb6 [feature](Outfile) select into outfile supports to export struct/map/array type data to orc file format (#24350)
We do not support nested complex type in this pr.
2023-09-21 20:15:18 +08:00
7630fe7b7b [bug](node)fix dense_rank function in partition sort node return wrong rows (#24727) 2023-09-21 19:13:30 +08:00
cbf2e913e8 [test](config) change desired_max_waiting_jobs in regression pipeline (#24710) 2023-09-21 17:27:52 +08:00
Pxl
bc0f5dfd8f [Chore](checks) move sonar config file to be(#24712)
move sonar config file to be
2023-09-21 15:16:26 +08:00
5b590bbfcf [feat](Nereids) add lambda func array_last and array_first (#24682) 2023-09-21 12:23:34 +08:00
Pxl
01b7cb8db7 [Bug](materialized-view) fix failed insert into mv meet default null (#24545)
fix failed insert into mv meet default null
2023-09-21 10:47:29 +08:00
b87ea68720 [Fix](statistics) Fix analyze olap table couldn't get partition names bug (#24696)
Call getPartitionNames to get all partitions while analyzing for olap table. Couldn't return NULL, otherwise analyze for olap table will do nothing.
2023-09-21 10:28:37 +08:00
0d20a61587 [fix](nereids)left outer join estimation (#24462)
A left join B on A.x=B.x and A.y=B.y
B.x and B.y make result tuple number scale out.
suppose A is scaled out by B.x N1 time, and scaled out by B.y N2 time, and N1 < N2.
we should choose N1 as the final scale out factor, not N2.

this pr impact on tpcds_sf100 59/17/29/25/47/40/54

before
query59 77295 75279 75230 75230
query17 22642 21566 21599 21566
query29 16508 16092 16006 16006
query25 20262 20571 21171 20571
query47 23571 23264 23107 23107
query40 3305 2849 3064 2849
query54 9052 8882 8715 8715
Total cold run time: 172635 ms
Total hot run time: 168044 ms

after
query59 56435 54717 53919 53919
query17 24167 22377 23237 22377
query29 16950 18325 16333 16333
query25 21478 22975 21358 21358
query47 24412 24611 23920 23920
query40 5491 4779 5176 4779
query54 8671 8664 8658 8658
Total cold run time: 157604 ms
Total hot run time: 151344 ms
2023-09-21 09:45:01 +08:00
4ca650f306 [improvement](jdbc catalog) Adjust function replacement order and add new function support (#24685) 2023-09-21 08:45:27 +08:00
5e82aebd25 [fix](fe-meta) fix paimon file catalog meta issue and replication num analysis issue (#24681)
Fix 2 issues:

The replication num analysis will throw exception when there is no BE alive when FE restart.
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = errCode = 2, detailMessage = replication num should be less than the number of available backends. replication num is 1, available backend num is 0
        at org.apache.doris.common.util.PropertyAnalyzer.analyzeReplicaAllocation(PropertyAnalyzer.java:1018) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.TableProperty.buildReplicaAllocation(TableProperty.java:483) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.TableProperty.read(TableProperty.java:529) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.OlapTable.readFields(OlapTable.java:1370) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Table.read(Table.java:383) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Database.readFields(Database.java:631) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.datasource.InternalCatalog.loadDb(InternalCatalog.java:3075) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Env.loadDb(Env.java:1764) ~[doris-fe.jar:1.2-SNAPSHOT]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_341]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_341]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_341]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_341]
        at org.apache.doris.persist.meta.MetaReader.read(MetaReader.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Env.loadImage(Env.java:1701) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.master.Checkpoint.doCheckpoint(Checkpoint.java:130) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.master.Checkpoint.runAfterCatalogReady(Checkpoint.java:79) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
Paimon File Catalog meta issue
Fix #23366
2023-09-21 08:41:03 +08:00
49d6627e31 [test](pipelinex) disable pipelinex case temporaraly #24699 2023-09-20 21:34:30 +08:00
2e85e0163d Revert "[feature](function) add json->operator convert to json_extract (#19899)" (#24679)
Revert "[feature](function) add json->operator convert to json_extract (#19899)"
because it conflict with lambda syntax
This reverts commit f54a068d82e88e8535f3ed55a4224886b752e46b.
2023-09-20 21:16:19 +08:00
294062d519 [bug](function) fix width bucket function return wrong result (#24673) 2023-09-20 20:37:22 +08:00
8aa39137fc [Bugfix](multi-catalog)If es hosts ends with "/", the index cannot be obtained. (#24272) 2023-09-20 20:35:14 +08:00
2a260be10c [improvement](jdbc catalog) when lower_case_table_names of jdbc catalog properties is set to true, use the real table name to query the jdbc data source (#24520) 2023-09-20 17:47:11 +08:00
81e65f4a12 [feature](function) Support SHA family functions (#24342) 2023-09-20 17:21:45 +08:00
5eb8fe3d6e [improvement](type) modify the inner type display of the Array/Map/Struct type (#24459)
In the old code, when using desc command to view the table schema
It will display as follows
```
ARRAY<TINYINT(4)>
ARRAY<SMALLINT(6)>
ARRAY<INT(11)>
ARRAY<BIGINT(20)>
ARRAY<LARGEINT(40)>
```
However, for normal integer type displays, the width is not displayed
So, I changed it to the following
```
ARRAY<TINYINT>
ARRAY<SMALLINT>
ARRAY<INT>
ARRAY<BIGINT>
ARRAY<LARGEINT>
```
2023-09-20 17:03:03 +08:00
a34757422c [regression](pipelineX) enable pipelineX test cases (#24670) 2023-09-20 16:32:03 +08:00
a946f99b8c [Fix](regression-test) fix regression-test of export parquet file format (#24450) 2023-09-20 15:41:49 +08:00
cf4143f3d1 [testcase](map) update regress test for array/map nested insert into nagetive cases (#24534) 2023-09-20 15:07:29 +08:00
7c32b2b4ed [Fix](broker load) broker load with or predicate error fix #24157
Co-authored-by: wangqingtao6 <wangqingtao6@jd.com>
2023-09-20 14:56:32 +08:00
fc12362a6d [feature-wip](arrow-flight)(step2) FE support Arrow Flight server (#24314)
This is a POC, the design documentation will be updated soon
2023-09-20 14:42:54 +08:00
a3361df7b9 [Feat](Nereids) support json and jsonb datatype (#24156)
Feature:
support jsonb and json type in nereids

Document:
this feature supports these two datatype in nereids optimizer like original planner, the sql reference is same as before
[JSON - Apache Doris](https://doris.apache.org/zh-CN/docs/dev/sql-manual/sql-reference/Data-Types/JSON)
2023-09-20 14:32:22 +08:00
e9435c14f8 [Improve](array-func)improve array union support multi params (#24327) 2023-09-20 14:29:48 +08:00
ca56921481 [docs](partition) Auto partition docs (#24574) 2023-09-20 14:28:23 +08:00