Commit Graph

14954 Commits

Author SHA1 Message Date
ffd4face00 [refactor](callonce) remove callonce usage in pipeline task to get full stack trace when core (#27331)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-11-21 19:42:39 +08:00
dea40e7095 [fix](Nereids): NullSafeEqual should be in HashJoinCondition (#27127)
Originally, we just put `EqualTo` in `HashJoinCondition`, we also need to allow `NullSafeEqual`
2023-11-21 19:08:14 +08:00
445484270b [fix](Nereids): fill miss slot in having subquery (#27177)
fill miss slot in having subquery.

such as 
```
select * from t group by k having max(k) in (select k from t2)
```

the max(k) should be push down aggregate
2023-11-21 18:54:58 +08:00
6fb57c2e1a [fix](docs) fix auto partition version label #27358 2023-11-21 18:38:48 +08:00
1ebb54afdc [fix](null equal) fix coredump of pushing eq_for_null (#27341) 2023-11-21 18:36:33 +08:00
f7a1c3edac [FIX](jsonb)fix jsonb is not in predict column #27325 2023-11-21 18:35:12 +08:00
e58983e59c [improvement](publish version) reduce publish version not exist log (#27332) 2023-11-21 18:00:01 +08:00
7e707f5d64 [fix](fe ut) Fix OlapQueryCacheTest failed (#27305)
1. 
```
java.lang.NullPointerException: null
        at org.apache.doris.catalog.Env.getCurrentSystemInfo(Env.java:793) ~[classes/:?]
        at org.apache.doris.qe.SimpleScheduler$UpdateBlacklistThread.run(SimpleScheduler.java:206) ~[classes/:?]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382]

java.lang.NullPointerException
        at org.apache.doris.qe.OlapQueryCacheTest.setUp(OlapQueryCacheTest.java:226)
```

2.
```
[ERROR] testSqlCacheKeyWithNestedViewForNereids  Time elapsed: 1.962 s  <<< FAILURE!
java.lang.AssertionError: SELECT command denied to user 'testCluster:testUser'@'192.168.1.1' for table 'internal: testCluster:testDb: appevent'
	at org.apache.doris.qe.OlapQueryCacheTest.parseSqlByNereids(OlapQueryCacheTest.java:579)
	at org.apache.doris.qe.OlapQueryCacheTest.testSqlCacheKeyWithNestedViewForNereids(OlapQueryCacheTest.java:1338)
```

3.
```
[ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 113.63 s <<< FAILURE! - in org.apache.doris.qe.OlapQueryCacheTest
[ERROR] testCacheModeTable  Time elapsed: 1.657 s  <<< ERROR!
java.lang.IllegalArgumentException: Value of type org.apache.doris.qe.QueryState incompatible with return type org.apache.doris.system.SystemInfoService of org.apache.doris.catalog.Env#getCurrentSystemInfo()
        at org.apache.doris.qe.OlapQueryCacheTest.setUp(OlapQueryCacheTest.java:156)
```
2023-11-21 16:47:57 +08:00
721282242b [pipelineX](bug) Fix potential memory access problem (#27326) 2023-11-21 14:36:25 +08:00
016dccb4b7 [fix](null_equal) fix wrong result and coredump of operator <=> (#27312)
* [fix](null_equal) fix wrong result and coredump of operator <=>

* fix
2023-11-21 14:32:03 +08:00
402095b85e [doc](fix) fix date trunc doc (#27317) 2023-11-21 14:14:37 +08:00
fcf7691e2f [pipeline](close) change the close order in pipeline engine (#27290) 2023-11-21 14:10:21 +08:00
b63f00987f [regression](invered index) add test null index case and fix fault injection index case (#27265) 2023-11-21 14:04:57 +08:00
dd2e9f682f [Fix](statistics)Fix analyze sql including key word bug (#27321)
Fix analyze sql including key word bug. Need use `` to wrap column names.
2023-11-21 13:15:37 +08:00
Pxl
cee8cc44e2 [Bug](insert)fix insert wrong data on mv when stmt have multiple values (#27297)
fix insert wrong data on mv when stmt have multiple values
2023-11-21 12:55:30 +08:00
d809bee46e [refactor](profilev2) add BlocksProduced RowsProduced counter #27291 2023-11-21 12:01:11 +08:00
3e8177bbbd Revert "[improvement](routine-load) add routine load rows check (#25818)" (#27304)
This reverts commit 5e3277e8fbc92f08344bc86e9b2c74033a8bbc01.
2023-11-21 11:28:05 +08:00
2548e27c97 [fix](memory) Fix work load group meaningless GC #27307 2023-11-21 09:59:21 +08:00
87a7de7660 [pipelineX](bug) Fix 'No broadcast buffer left' (#27298) 2023-11-20 22:37:21 +08:00
c0f22e8feb [FIX](complextype)fix struct nested complex collection type and and regresstest (#26973) 2023-11-20 22:29:12 +08:00
840f3b6439 [pipelineX](dependency) Wake by task by read dependency (#27260) 2023-11-20 22:00:59 +08:00
0b459e50fb [fix](partial update) keep case insensitivity and use the columns' origin names in partialUpdateCols in origin planner (#27223)
close: #27161
2023-11-20 21:16:28 +08:00
fec94b7278 [feature](Nereids): use session variable to enable rule (#27036) 2023-11-20 20:23:24 +08:00
20d7ab061b [PipelineX](complex type pushdown) do not push down complex type to storage layer based on PR#25007 (#27278) 2023-11-20 18:47:59 +08:00
19b1d5365c [minor](stats) rename stats related session variable name #26936 2023-11-20 18:13:12 +08:00
273cbfc36c [fix](inverted index) reset fs_writer to nullptr before throw exception (#27202) 2023-11-20 17:40:56 +08:00
7164b7cebb [case](regression) add read json by line test case (#26670)
Co-authored-by: qinhao <qinhao@newland.com.cn>
2023-11-20 17:33:40 +08:00
80c75b6da4 [fix](schema change) fix bug of query failure after rename column (#26300) 2023-11-20 16:54:40 +08:00
bd2e8239f8 [fix](broker) not print broker request detail default (#27126)
if broker detail contails ak/sk or some sensitive information, not print these info default in log
---------
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2023-11-20 16:41:47 +08:00
34c3cde0de Revert "[feature-wip](catalog) support deltalake catalog step1-metadata (#22493)" (#27095)
This reverts commit 5b641ebd40fff71e632ee9be4ede58b744b602b9.

Currently, Deltalake Catalog is not a usable feature. We will continue to implement it in the datalake plug-in system in the future, so we will delete it from the FE code for now.
2023-11-20 16:10:33 +08:00
d939903753 [improvement](statistics)Use count as ndv for unique/agg olap table single key column (#27186)
Single key column of unique/agg olap table has the same value of count and ndv, for this kind of column,
don't need to calculate ndv, simply use count as ndv.
2023-11-20 15:49:08 +08:00
6ed0be8e3c [refactor](profilev2) unify the counter name in shuffle operator and normal operator (#27267)
using blocksproduced and rowsproduced to unify the counter name in DataStreamSender and other exec node, or exchange operator and other operators.
blocks produced and rows produced are more easy to understand.
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-11-20 14:21:39 +08:00
459f75073f [pipelineX](dependency) remove OrDependency (#27242) 2023-11-20 13:05:34 +08:00
Pxl
3f3f81f474 [Documentation](materialized-view)updatee doc about materialized-view's supported aggregate function (#27119)
updatee doc about materialized-view's supported aggregate function
2023-11-20 10:58:33 +08:00
add6bdb240 [fix](multi-catalog)add the max compute fe ut and fix download expired (#27007)
1. add the max compute fe ut and fix download expired
2. solve memery leak when allocator close
3. add correct partition rows
2023-11-20 10:42:07 +08:00
8c9a22cc4f [fix](tablesample) Fix computeSampleTabletIds NullPointerException (#27165)
```
2023-11-05 22:52:01,914 WARN (mysql-nio-pool-167|1655) [StmtExecutor.analyze():992] Analyze failed. stmt[222101, ec1adf0f9f7b405d-8cb71a522bc50b82]
java.lang.NullPointerException: null
        at org.apache.doris.planner.OlapScanNode.computeSampleTabletIds(OlapScanNode.java:952) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.OlapScanNode.init(OlapScanNode.java:548) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.SingleNodePlanner.createScanNode(SingleNodePlanner.java:2064) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:2213) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:1244) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:266) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:189) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.OriginalPlanner.createPlanFragments(OriginalPlanner.java:160) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.OriginalPlanner.plan(OriginalPlanner.java:101) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.analyzeAndGenerateQueryPlan(StmtExecutor.java:1141) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:975) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:673) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:451) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:422) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:435) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:583) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:834) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_333]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_333]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_333]
2023-11-05 22:52:01,914 WARN (mysql-nio-pool-167|1655) [StmtExecutor.executeByLegacy():776] execute Exception. stmt[222101, ec1adf0f9f7b405d-8cb71a522bc50b82]
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Unexpected exception: null
        at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:993) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:673) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:451) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:422) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:435) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:583) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:834) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_333]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_333]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_333]
```
2023-11-20 10:03:47 +08:00
767b5a1770 [test](external_table_p0)append log in external_table_p0 for debug unknown table case (#27212) 2023-11-19 22:47:32 +08:00
23ab3333a0 [chore](thirdparty) Fix link errors of librdkafka (#27247)
Exclude coverage symbol from librdkafka's version script generator to
avoid compatibility issues with recent linkers such as lld-17.
2023-11-19 21:22:26 +08:00
58aefbc33f [feature](clucene) coverage compilation option added (#27162) 2023-11-19 05:33:08 -06:00
c7cefe27d5 [pipelineX](broadcast) Add error logs (#27243) 2023-11-19 17:02:00 +08:00
70e070182f [feature](executor)Make workload group property not required (#27229)
* Make workload group property not required

* remove useless  UT
2023-11-19 17:01:51 +08:00
ab16d0f473 [com](github) add auto replay on new PR to show how to run pipeline (#27219) 2023-11-19 17:01:38 +08:00
febd60c75f [fix](join) incorrect result of left join with other conjuncts (#27238) 2023-11-19 15:36:39 +08:00
836cda65d8 [refactor](profilev2) split merged profile to a single runtime profile to make the logic more clear (#27184) 2023-11-19 13:21:50 +08:00
hat
61fe545e48 [fix][chore](clang-tidy) Fix order of tidy Checks (#27237)
Clang-tidy add checks based on the order of `Checks` field.
Here the `-*` removes the `clang-diagnostic-*,clang-analyzer-*` before.
2023-11-19 13:06:01 +08:00
7b817197b2 [pipelineX](sink) simplify result sink dependency (#27226) 2023-11-19 12:42:10 +08:00
b560d863c2 [fix](dynamic) Fix error reporting when dynamic partition properties contain incorrect attributes (#25373) 2023-11-19 09:51:57 +08:00
329abc3452 [feature](nereids) runtime filter prune when column stats are not available (#27099)
1. prune rf when column stats are not available
2. print rf in "explain shape plan", both join side and apply side
3. add regression case to check plan shape/rf/rf prune for tpch_sf1000 (stats are not available)
2023-11-18 19:32:33 +08:00
b42828cf69 [fix](window_function) min/max/sum/avg should be always nullable (#27104)
Co-authored-by: starocean999 <40539150+starocean999@users.noreply.github.com>
2023-11-18 18:41:42 +08:00
2f41e0c823 [FIX](complextype)fix information schema for complex type (#27203)
when we select in information schema , here do not show complex type information
2023-11-18 11:32:32 +08:00