Commit Graph

18197 Commits

Author SHA1 Message Date
f52067415b Improve analyze mv/mtmv wait row count report logic. (#33695) 2024-04-17 23:42:13 +08:00
ff8cb3cc43 [Fix](executor)Fix routine load failed when can not find group (#33596) 2024-04-17 23:42:13 +08:00
b44fed8dc2 [fix](restore) Reset index id for restore (#33648) 2024-04-17 23:42:13 +08:00
11f7d11a96 [branch-2.1](regression-test) fix test_chema_change_fail (#33753) #33788 2024-04-17 23:42:13 +08:00
5734e2bd30 [opt](meta-cache) refine the meta cache (#33449) (#33754)
bp #33449
2024-04-17 23:42:13 +08:00
6976d019a3 [opt](inverted index) topn opt reads only limit number of records (#33665) 2024-04-17 23:42:13 +08:00
2854048eb5 fix compile 2024-04-17 23:42:13 +08:00
5053ac2b59 [improve](CI)Core modules require maintainer review (#32468) 2024-04-17 23:42:13 +08:00
43974a2334 (Fix)(nereids) modify create view privilege check error message (#33669) 2024-04-17 23:42:13 +08:00
db846709d2 [opt](Nereids) auto fallback when meet udf override (#33708) 2024-04-17 23:42:13 +08:00
81f7c53bad [fix](Nereids) could not query variant that not from table (#33704) 2024-04-17 23:42:13 +08:00
1c025c0488 [docker](hive) add hive3 docker compose and modify scripts (#33115)
add hive3 docker compose from:
big-data-europe/docker-hive#56
2024-04-17 23:42:13 +08:00
22a6b1d3f5 [feature](function) support hll functions hll_from_base64, hll_to_base64 (#32089)
Issue Number: #31320 

Support two hll functions:

- hll_from_base64
Convert a base64 string(result of function hll_to_base64) into a hll.
- hll_to_base64
Convert an input hll to a base64 string.
2024-04-17 23:42:13 +08:00
3096150d1b [feature](agg) support aggregate function group_array_intersect (#33265) 2024-04-17 23:42:13 +08:00
07a8f44443 [improvement](spill) improve config and fix spill bugs (#33519) 2024-04-17 23:42:13 +08:00
3f267e36d1 [fix](nereids)InSubquery's withChildren method lost typeCoercionExpr (#33692) 2024-04-17 23:42:13 +08:00
b07e0a2f06 [FIX](cast)fix full/right out join for cast array (#33475)
in some case, we has code
```
        if (_join_op == TJoinOp::RIGHT_OUTER_JOIN || _join_op == TJoinOp::FULL_OUTER_JOIN) {
            _probe_column_convert_to_null = _convert_block_to_null(*input_block);
        }
```
then do next function like cast , but in function cast we assume block column is same with from_type.which will make status error
2024-04-17 23:42:13 +08:00
2890f6c3cf [opt](Nereids) date literal support basic format with timezone (#33662) 2024-04-17 23:42:13 +08:00
11266dd9b8 [minor](Nereids): remove useless override (#33651) 2024-04-17 23:42:13 +08:00
16e9eb3b05 [fix](analyze) avoid java.util.ConcurrentModificationException (#33674)
```
java.util.ConcurrentModificationException: null
        at java.util.TreeMap$ValueSpliterator.forEachRemaining(TreeMap.java:3226) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.apache.doris.statistics.AnalysisManager.findShowAnalyzeResult(AnalysisManager.java:552) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.statistics.AnalysisManager.showAnalysisJob(AnalysisManager.java:533) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ShowExecutor.handleShowAnalyze(ShowExecutor.java:2772) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ShowExecutor.execute(ShowExecutor.java:447) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.handleShow(StmtExecutor.java:2738) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:1010) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:624) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:526) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:333) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:228) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:176) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:205) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:258) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
```

Due to the `Collections.synchronizedNavigableMap`'s java doc:

```
In order to guarantee serial access, it is critical that all access to the backing navigable map is accomplished through the returned navigable map (or its views).
It is imperative that the user manually synchronize on the returned navigable map when traversing any of its collection views, or the collections views of any of its subMap, headMap or tailMap views, via Iterator, Spliterator or Stream
```
2024-04-17 23:42:13 +08:00
ca728a2405 [feature](proc)Add table's indexes info in show proc interface (#33438)
1. Add show proc `/dbs/db_id/table_id/indexes` impl
2. Remove index_id in `show index from table`
3. Add test cases

---------

Co-authored-by: Luennng <luennng@gmail.com>
2024-04-17 23:42:13 +08:00
dac2829194 [fix](routine-load) fix data lost when FE leader change (#33678) 2024-04-17 23:42:13 +08:00
53a3d6c154 [fix](testcase) fix miss used global variables in index testcases (#33293) 2024-04-17 23:42:13 +08:00
775022c204 [refactor](pipelineX) Reduce prepare overhead (PART II) (#33681) 2024-04-17 23:42:13 +08:00
59de97be5e [improvement](mow) Add profile for delete_bitmap get_agg function (#33576) 2024-04-17 23:42:13 +08:00
d6a63b9789 [Update](cloud) skip show data size assertion in cloud mode (#33677) 2024-04-17 23:42:13 +08:00
d15981abd2 [Enhencement](Nereids) add rule of agg(case when) to agg(filter) (#33598) 2024-04-17 23:42:13 +08:00
1fba73eea4 [fix](fe) Fix finalizeCommand sendAndFlush NullPointerException (#33420) 2024-04-17 23:42:13 +08:00
fe3b6824b3 [case](regression) Add backup temp partition case (#33646) 2024-04-17 23:42:13 +08:00
8e38549a92 [fix](nereids) Use correct PREAGGREGATION in agg(filter(scan)) (#33454)
1. set `PreAggStatus` to `ON` when agg key column by max or min;
2. #28747 may change `PreAggStatus` of scan, inherit it from the previous one.
2024-04-17 23:42:13 +08:00
d18f5e2544 [refactor](refresh-catalog) refactor the refresh catalog code (#33653)
To unify the code.
In previous, we do catalog refresh in `CatalogMgr`, but do
database and table refresh in `RefreshMgr`, which is very confusing.

This PR move all `refresh` related code from CatalogMgr to RefreshMgr.

No logic is changed in this PR.
2024-04-17 23:42:12 +08:00
4863167f90 [refactor](pipelineX) Reduce prepare overhead (PART I) (#33550) 2024-04-17 23:42:12 +08:00
e0ec2da29b [fix](routine-load) fix get kafka offset timeout may too long (#33502) 2024-04-17 23:42:12 +08:00
Pxl
341cb40693 [Chore](log) adjust output order on PrintInstanceStandardInfo and reduce warning log when rpc finished (#33652)
adjust output order on PrintInstanceStandardInfo and reduce warning log when rpc finished
2024-04-17 23:42:12 +08:00
466b9f35d5 [fix](nereids)EliminateGroupBy should keep the output's datatype same as old ones (#33585) 2024-04-17 23:42:12 +08:00
c15ac3ffca [testcases](auto-partition) fix data sync (#33635)
fix data sync
2024-04-17 23:42:12 +08:00
7b16cb5a4c [feature](inverted index) add slop functionality to match_phrase (#33225)
https://github.com/apache/doris-website/pull/553 doc
2024-04-17 23:42:12 +08:00
f6af79c0ed [fix](catalog) Remove unexpected cleanup when reading jdbc data (#33529) 2024-04-17 23:42:12 +08:00
7659b1aa67 [opt](Nereids) prefer slot type to support delete task better (#33559) 2024-04-17 23:42:12 +08:00
5b616da543 [refine](Operator) When _stop_emplace_flag is not set to true, perform batch processing on the block. (#33173) 2024-04-17 23:42:12 +08:00
e53a76d75b [fix](planner) fix bug of InlineViewRef's tableNameToSql method (#33575) 2024-04-17 23:42:12 +08:00
d000658a9b [fix](nereids) ExtractAndNormalizeWindowExpression should only normalize alias in output (#33527) 2024-04-17 23:42:12 +08:00
87e6c94851 [docker](script)add --grace to be_prestop.sh (#33599) 2024-04-17 23:42:12 +08:00
b2face0d20 [feature](Nereids): date literal suppose Zone (#33534)
support
```
'2022-05-01 01:02:55+02:30
'2022-05-01 01:02:55Asia/Shanghai
```
2024-04-17 23:42:12 +08:00
3df8f0cad8 [improve](move-memtable) add more info in LoadStreamStub errors (#33618) 2024-04-17 23:42:12 +08:00
46a258dc85 [improvement](binlog)Support inverted index format v2 in CCR (#33415) 2024-04-17 23:42:12 +08:00
690bf54346 [runtime filter](fix) Fix wrong results caused by IN_OR_BLOOM filter (#33701) (#33715) 2024-04-17 23:42:12 +08:00
Pxl
918f7225b4 [Bug](runtime-filter) make need_local_merge unrelated with broadcast and support merge on bitmap_filter (#33664) 2024-04-17 23:42:12 +08:00
f8acb1ee2e [fix](merge-on-write) schema change may cause mow duplicate key (#33536) (#33660) 2024-04-17 23:42:12 +08:00
06a155abb0 [branch-2.1](cherry-pick) Pick some partial-update PR from master (#33639)
* [Fix](partial-update) Fix partial update fail when the datetime default value is 'current_time' (#32926)

* Problem: When importing data that includes datetime with a default value of current time for partial column updates, the import fails.
Reason: Partial column updates do not handle the logic for datetime default values.
Solution: During partial column updates, when the default value is set to current time, read the current time from the runtime state and write it into the data.

* [Enhancement](partial update)Add timezone case for partial update timestamp #33177

* [fix](partial update) Support partial update when the date default value is 'current_date'. This PR is a extension of PR #32926. (#33394)
2024-04-17 23:42:12 +08:00