Commit Graph

14397 Commits

Author SHA1 Message Date
c715facafa [fix](window_function) window function first_value/last_value should be always nullable (#26014) 2023-10-27 20:53:48 +08:00
5e884bc065 [Fix](bitmap index) fix is_like_predicate for bitmap index evaluate (#25961) 2023-10-27 20:23:42 +08:00
d953e5c8f4 [feature](Nereids): Convert topn(x, 1) to max(x) (#26004) 2023-10-27 18:45:21 +08:00
3beba1764c [Test](multi-catalog) Add tpcds sf100 hive shape. (#25639)
Add tpcds sf100 hive shapes.

Disable query64 temporarily because it is not same with emr cluster after collecting metadata by analyze table xxx.
And the root cause need to analyze, will enable in future PR.
2023-10-27 18:39:29 +08:00
606223ab62 Revert "[refactor](pipeline) simplify runtime state ctor (#25995)" (#26029)
This reverts commit a01922cdc55e2b3a63d9a9aafb38ac5ed64c6dd3.
2023-10-27 18:15:30 +08:00
2464a22ac5 [fix](meta) fix follower sync image timeout after checkpoint. (#26003)
The image file of our cluster reaches 2.3G. After the checkpoint, Followers synchronize the image timeout, resulting in the continuous increase of the bdb directory.

related pr: #25768
2023-10-27 16:58:42 +08:00
165957658a [fix](Nereids) could not run multi group_concat distinct (#25851)
could not run multi group_concat distinct with more than one parameters.
This bug is not just for group_concat, but we usually use literal as
parameters in group_concat. So group_concat brought the problem to light.

In the original logic, we think only distinct aggregate function with
zero or one parameter could run in multi distinct mode. But it is wrong.
We could process all distinct aggregate function with not more than one
input slots.

Think about sql:
```sql
SELECT
  group_concat(distinct c1, ','), group_concat(distinct c2, ',')
FROM t
GROUP BY c3
```
2023-10-27 03:42:00 -05:00
11e04f76fb [Enhancement](submodule) print current submodule commit id when building (#26017) 2023-10-27 16:28:27 +08:00
340a37116f [cleanup](load) remove unused dynamic schema code (#25932) 2023-10-27 16:21:40 +08:00
a01922cdc5 [refactor](pipeline) simplify runtime state ctor (#25995) 2023-10-27 15:45:29 +08:00
cad3d8fb99 [doc](complex type) add docs for complex and nested types (#25865)
Add docs for complex and nested types.
2023-10-27 15:04:53 +08:00
d57fc263cf [fix](Nereids) support create dup table without key (#25954) 2023-10-27 01:57:39 -05:00
4aebe879a6 [feature](Nereids) support complex project in graph simplifier (#26002)
Reject the edge which has an alias when ordering edge
2023-10-27 14:38:54 +08:00
a6d64c6b25 [refine](profilev2) format profilev2 (#25963) 2023-10-27 14:26:38 +08:00
4ee24d8cb1 [FIX](upgrade)fix upgrade for predict column delete collection type will make core #26006 2023-10-27 14:25:45 +08:00
46d40b1952 [refactor](executor)Remove empty group logic #26005 2023-10-27 14:24:41 +08:00
e3dbbed47c [fix](Nereids) query failed when fold constant on be failed (#25982)
we translate expression to legacy one when do fold constant on BE.
some times, we generate invalid expression that cannot be tranlsated.
So, we should catch translate exception to avoid query failed.
2023-10-27 01:22:08 -05:00
deb1ad8264 [enhancement](Nereids) add mode no_backslash_escapes (#25822)
add mode [no_backslash_escapes](https://dev.mysql.com/doc/refman/8.0/en/sql-mode.html)
2023-10-27 00:43:32 -05:00
ae8e6fbe59 [Fix](inverted index) fix empty array index writer bug (#25984) 2023-10-27 11:27:44 +08:00
cedab51676 [enhancement](UT) add unit test cases about bitmap (#25867)
* [fix](bitmap) incorrect result of operator ==

* [enhancement](UT) add unit test cases about bitmap
2023-10-27 11:27:14 +08:00
6a62835f7e [fix](docker) Add docker ignore files and change docker default sourc… (#25696) 2023-10-27 11:23:21 +08:00
a3ac8b98b8 [fix](planner)nvl should match function signature in the same way as ifnull (#25849) 2023-10-27 11:13:30 +08:00
8206252e0a [Performance](Nereids): refactor GroupMatching (#25960)
Refactor GroupMatching to improve performance.
2023-10-27 10:56:42 +08:00
bac523956f [Update](clucene) update clucene version to latest (#25981) 2023-10-27 10:42:56 +08:00
0eb86919cc [fix](block-convertor) clear filter bitmap per batch to get the correct filtered rows (#25989) 2023-10-27 10:40:07 +08:00
7a69bdd1cd [regression](fe) Add more regression test for FE (#25976) 2023-10-27 10:27:22 +08:00
d0eaf95977 [cases](regression-test) Add alter table properties and alter/rename view test (#25971) 2023-10-27 10:27:06 +08:00
fa06c7517a [feature](meta) Add thrift rpc to get db/table/backend meta (#25943)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-10-27 10:12:58 +08:00
b32557e195 [fix](stats) Fix potential NPE when doing auto analyze (#25908) 2023-10-26 20:47:57 -05:00
f737ba815d [fix](nereids) push down subquery exprs in non-distinct agg functions (#25955) 2023-10-26 20:36:10 -05:00
c86fad7cbd [Fix](orc-reader) Fix orc decimal128 scale issue. (#25977) 2023-10-26 08:50:18 -05:00
d9a3aa5c06 [pipelineX](improvement) improve local shuffle (#25964) 2023-10-26 21:42:43 +08:00
36be7dcc13 [enhancement](regression) add normal query before date convert for debug (#25970)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2023-10-26 21:37:28 +08:00
69015df767 [Fix](inverted index) reorder ConjunctionQuery deconstruct order (#25972) 2023-10-26 08:34:09 -05:00
1e39636fbf [fix](nereids) fix wrong result precision for add/sub (#25751) 2023-10-26 08:28:34 -05:00
9faa6e083b [fix](udf) avoid exception when fail to find udf in replay logic (#25965)
When replaying drop function edit log, the function may not be found, causing runtime exception and
FE will fail to start.
The function SHOULD be exist, but the reason is still unknown.
I change the logic to NOT throw exception if function is not found.
This is a workaround to make sure FE can start, and add some log for later debug.
2023-10-26 08:19:13 -05:00
a2a157fcde [ci](pipeline) add required of "P0 Regression PipelineX (Doris Regression)" (#25967) 2023-10-26 19:17:12 +08:00
678dc366e0 [fix](export) fix timeout property not work for export job (#25913)
Co-authored-by: caiconghui1 <caiconghui1@jd.com>
2023-10-26 18:51:57 +08:00
2679fa4ea7 [improvement](tablet clone) furthur repair replicas should be check even if they are versions catchup (#25551) 2023-10-26 18:14:40 +08:00
c3527672a5 [refactor & pipelineX][pick fix] Pick fix of predicate pushdown to pipelineX (#25953)
Co-authored-by: JackDrogon <jack.xsuperman@gmail.com>
2023-10-26 18:04:43 +08:00
c1d64a7128 [Feature](datatype) Add IPv4/v6 data type for doris (#24965) 2023-10-26 17:33:28 +08:00
9705ec760c [fix](regression-test) use correct dataset for unique_with_mow_p2 (#25653) 2023-10-26 16:54:27 +08:00
642c701819 [doc](partial update) add flink-connector docs (#25843) 2023-10-26 16:53:06 +08:00
de06a2f8b7 [minor](test) remove useless date case (#25941)
remove case
```sql
select datediff('10000-10-31', '2010-10-15')
```

because Nereids throw exception when cast '10000-10-31' to date,
but legacy planner return null on it
2023-10-26 03:16:11 -05:00
1ba8a9bae4 [feature-wip](executor)Fe send topic info to be (#25798) 2023-10-26 15:52:48 +08:00
2229d82acd [fix](Nereids) should not put bound expr into unbound group by list (#25938)
we put bound expr into unbound group by list by mistake.
This will lead to bind twice on some exprssion.
Since binding is not idempotent, below exception will be thrown for sql

```sql
select k5 / k5 as nu, sum(k1) from test group by nu order by nu nulls first
```

```
Caused by: org.apache.doris.nereids.exceptions.AnalysisException: Input slot(s) not in child's output: k5#5 in plan: LogicalProject[176] ( distinct=false, projects=[(cast(k5#5 as DECIMALV3(16, 10)) / k5#5) AS `nu`#14, sum(k1)#15], excepts=[] ), child output is: [nu#16, sum(k1)#15]
plan tree:
LogicalProject[176] ( distinct=false, projects=[(cast(k5#5 as DECIMALV3(16, 10)) / k5#5) AS `nu`#14, sum(k1)#15], excepts=[] )
+--LogicalAggregate[168] ( groupByExpr=[nu#16], outputExpr=[nu#16, sum(k1#1) AS `sum(k1)`#15], hasRepeat=false )
   +--LogicalProject[156] ( distinct=false, projects=[k1#1, (cast(k5#5 as DECIMALV3(16, 10)) / k5#5) AS `nu`#16], excepts=[] )
      +--LogicalOlapScan ( qualified=default_cluster:regression_test_nereids_syntax_p0.test, indexName=test, selectedIndexId=503229, preAgg=OFF, Aggregate function sum(k1) contains key column k1. )
    at org.apache.doris.nereids.rules.analysis.CheckAfterRewrite.checkAllSlotReferenceFromChildren(CheckAfterRewrite.java:108) ~[classes/:?]
```
2023-10-26 02:31:20 -05:00
bc606859c7 [enhancement](regression-test) add test for test_avg (#25892) 2023-10-26 14:33:49 +08:00
d6c64d305f [chore](log) Add log to trace query execution #25739 2023-10-26 14:09:25 +08:00
6ce1087c23 [enhancement](stats) Limit analyze info count (#25576)
Each analyze job info and task info would not exceed 20000 after this PR. User could adjust this by FE conf param: analyze_record_limit
2023-10-26 13:46:28 +08:00
d896debbe7 [fix](stats) Escape string when insert stats (#25815)
Co-authored-by: AKIHA <cyborgz1999@example.com>
2023-10-26 13:39:41 +08:00