Commit Graph

16239 Commits

Author SHA1 Message Date
d50c8b6d3a [Improvement](nereids) Query rewrite by mv support bitmap_union and bitmap_union_count roll up (#29418)
Query rewrite by mv support bitmap_union and bitmap_union_count roll up, aggregate functions which supports roll up is listed as following:

| 查询中函数            | 物化视图中函数      | 函数上卷后              |
|------------------|--------------|--------------------|
| max              | max          | max                |
| min              | min          | min                |
| sum              | sum          | sum                |
| count            | count        | sum                |
| count(distinct ) | bitmap_union | bitmap_union_count |
| bitmap_union | bitmap_union | bitmap_union|
| bitmap_union_count | bitmap_union | bitmap_union_count |

this depends on  https://github.com/apache/doris/pull/29256
2024-01-12 11:44:21 +08:00
0a55376a48 [fix](regression-test) fix non stable case test_bitmap_index (#29592)
Wrong wait getJobState make this case not stable.

Master && Branch2.0 both have this problem.
2024-01-12 11:44:21 +08:00
161134913d [fix](regression-test) fix non stable case test_unique_table_like (#29643)
test_unique_table.groovy and test_unique_table_like.groovy both use database test_unique_db.

If they run at the same time, we may got the following errors:

java.sql.SQLException: errCode = 2, detailMessage = There are still some transactions in the COMMITTED state waiting to be completed. The database [default_cluster:test_unique_db] cannot be dropped. If you want to forcibly drop(cannot be recovered), please use "DROP database FORCE".

Both Master & Branch2.0 have this problem.
2024-01-12 11:44:21 +08:00
1d834571ab [doc](fix) K8s doc modify catalog and add helm doc (#29604) 2024-01-12 11:44:21 +08:00
87023d3b7a [Fix](inverted index) fix memory leak in inverted index when encountering fault (#29676) 2024-01-12 11:44:21 +08:00
1ca4d8bf10 [improve](docs) add Flink write metrics doc (#29578) 2024-01-12 11:44:21 +08:00
75631c89fa [Fix](thirdparty) fix that BE can not start when use JDK17 (#29570)
An error occurred when starting BE with JDK17

```java
Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "process reaper"
```

This error occurs when BE's java code calls Runtime.exec() to fork the child process.
It turned out that Doris was calling the `glog` library in the C++ layer to cause this problem.


The solution comes from: https://github.com/google/glog/issues/975
2024-01-12 11:44:21 +08:00
0b51532dfc [doc](release) update release process related to maven (#29467) 2024-01-12 11:44:21 +08:00
9f73f77d12 [fix](pipeline) incorrect result caused by missing output block of union operator (#29677) 2024-01-12 11:44:21 +08:00
aa4de6f39a (feature)[pipelineX]Make operator_id negative in pipelineX (#29649)
"operator_id" should be invisible, but the local shuffle is a planned operator in the BE (Backend), without a plan node ID. We use it in profiles and other places, and there might be duplicates. Therefore, we switch it to a negative number here to distinguish it as a plan node ID.
2024-01-12 11:44:21 +08:00
2ca90b2bf1 [Refactor](dialect) Add sql dialect converter plugins (#28890)
The current logic for SQL dialect conversion is all in the `fe-core` module, which may lead to the following issues:
- Changes to the dialect conversion logic may occur frequently, requiring users to upgrade the Doris version frequently within the fe-core module, leading to a longer change cycle.
- The cost of customized development is high, requiring users to replace the fe-core JAR package.

Turning it into a plugin can address the above issues properly.
2024-01-12 11:44:20 +08:00
28dca66c06 Fix BE UT macOS compile (#29663) 2024-01-12 11:44:20 +08:00
40badbf5c5 Fix analyze empty external NPE bug. (#29675) 2024-01-12 11:41:21 +08:00
54d2528c69 [Fix](Nereids) fix fe ut failed cause of getting statement context (#29683)
Problem:
fe ut failed cause of null pointer error
Cause:
fe ut getting statement context from connection context failed
Resolved:
add null pointer judgement
2024-01-12 11:41:06 +08:00
a2da434e3b [refactor](Nereids): refactor PredicatePropagation & support to infer Equal Condition (#29644) 2024-01-12 11:40:57 +08:00
8fc9c18c85 [improvement](jdbc catalog) Put the jdbc connection pool parameters into catalog properties (#29195) 2024-01-12 11:40:28 +08:00
30e46ee5ad [Fix](Job)Fixed the problem of not deleting JOB during DROP JOB metadata playback (#29543) 2024-01-12 11:40:19 +08:00
78fc38f53e [typo](docs) fix the unit of default value of thrift_connect_timeout_seconds (#29626) 2024-01-12 11:40:08 +08:00
3cd1c7745a [fix](jdbc catalog) Fix the precision of decimal type mapping to 0 (#29407) 2024-01-12 11:39:57 +08:00
c10bcb666d [Fix](Nereids) change log level of warning of converting error to debug (#29660)
Co-authored-by: libinfeng <libinfeng@selectdb.com>
2024-01-12 11:39:49 +08:00
eea657a610 [rf](nereids)prune rf for external db according to jump count (#29634)
* prune some rf for external db
2024-01-12 11:37:16 +08:00
c9e2f1934a [opt](file cache) make the cache queue percentage configurable (#29537) 2024-01-12 11:36:58 +08:00
97ed06a92c [regression-test](Variant) fix unstable case (#29648) 2024-01-12 11:36:45 +08:00
b59a8c9365 [feature](Nereids): refresh view hypergraph after inferring join (#29469) 2024-01-12 11:36:21 +08:00
0126b09c11 [fix](Nereids): fix shape of eliminate_inner regression test (#29613) 2024-01-12 11:36:03 +08:00
971bc804ac [fix](Nereids) update and delete may produce exprs with same exprid (#29656) 2024-01-12 11:35:49 +08:00
847898bf26 [fix](Nereids) delete using should support sql without where (#29518) 2024-01-12 11:35:29 +08:00
ddaa645a4f [improvement](statistics) Force to use zonemap for collecting string type min max. (#29631)
Force to use zonemap for collecting string type min max.
String type is not using zonemap for min max, because zonemap value at BE side is truncated at 512 bytes which may cause the value not accurate. But it's OK for statisitcs min max, and this could also avoid scan whole table while sampling.
2024-01-12 11:34:07 +08:00
9791a63212 [opt](match_phrase) Optimizing match_phrase with a new algorithm (#29444)
Based on the latest Lucene algorithm
2024-01-12 11:33:44 +08:00
53f1521308 [feature](inverted index)Support failover when index compaction failed (#29553) 2024-01-12 11:33:15 +08:00
9ef4e49307 [bugfix](scannerdeadloop) there is a dead loop in scanner ctx (#29794)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-01-11 16:47:54 +08:00
223e466514 [fix](insert-into) fix insert into lose data (#29802) 2024-01-11 16:47:25 +08:00
01a91a482e [pipelineX](fix) fix StreamingAggSource crash due to empty data block (#29769) 2024-01-10 17:13:24 +08:00
443b79d6ba [pipelineX](bug) Fix correctness problem using multiple BE (#29765) 2024-01-10 17:13:13 +08:00
fae7a395de [Bug](FS) Fix S3FileSystem::file_size_impl #29720 2024-01-10 10:34:10 +08:00
3675e0302c [fix](nereids) generate correct order for runtime filter when contains NullSafeEquals hash condition (#29726)
Be do not support RF for NullSafeEquals, so fe not generate RF for them.
However, after we support NullSafeEquals as Hash join condition,
the order of RF is wrong when generating RF in FE. this PR fix it.
2024-01-10 10:33:45 +08:00
e2ccca6290 [pipelineX](bug) Fix runtime filter timeout (#29738) 2024-01-10 10:32:57 +08:00
767de7afe8 Revert "[feature](pipelineX) control exchange sink by memory usage (#28814)" (#29652)
This reverts commit e326ebb63e4e07d8ee6595561ab19dc5d411f592.
2024-01-08 21:48:51 +08:00
59d7f64360 [Fix](Nereids) fix pipelineX distribute expr list with child output expr ids (#29621) 2024-01-08 10:46:27 +08:00
Pxl
e556536de1 [Refactor](join) split SetHashTableVariants out from HashTableVariants (#29519)
split SetHashTableVariants out from HashTableVariants
2024-01-08 10:37:00 +08:00
0bdd007926 [improve](insert-into) add log when instance mark and done (#29636) 2024-01-08 10:11:12 +08:00
c58d18147b [refactor](resultwriter) rename append_block to write method is more meaningful (#29635)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-01-08 00:00:33 +08:00
c497f749ce [debug](timeout) debug select timeout (#29627)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-01-07 19:54:02 +08:00
8e49aab7ba [fix](case) Update test_stream_load.groovy (#29551)
add sql "sync" after streamload
2024-01-07 19:53:17 +08:00
17cf4ab2c1 [case](regression) streamload publish timeout (#29457)
Co-authored-by: qinhao <qinhao@newland.com.cn>
2024-01-07 19:50:16 +08:00
1ea51e9f20 [Feature](group commit) Support table property "group commit data bytes" (#29484) 2024-01-07 19:46:42 +08:00
2d89b7aed4 [fix](tablet sched) disable disk balance for single replica (#29576) 2024-01-07 19:21:42 +08:00
eb2b22bff1 [improve](cooldown) skip empty tablet (#29620) 2024-01-07 18:57:06 +08:00
0b731800a0 [enhancement](group_commit) refector wal manager code (#29560) 2024-01-07 18:54:41 +08:00
e7b5f79664 [Fix](group commit) Fix wal mem back pressure fault injection case (#29625) 2024-01-07 18:31:31 +08:00