Commit Graph

15002 Commits

Author SHA1 Message Date
93cfdffb75 [regression test](routine test) add case for exec_mem_limit (#27308) 2023-11-23 14:25:54 +08:00
7a75f8c380 [improve](move-memtable) set brpc streaming params in config (#27442) 2023-11-23 14:14:43 +08:00
dbbed113cf [feature](mtmv)(4)MTMV extends Olap (#26645) 2023-11-23 14:10:36 +08:00
4fc638c98c [fix](memory) Fix make_top_consumption_snapshots heap-use-after-free #27434 2023-11-23 13:59:53 +08:00
75d1594d11 [pipelineX](minor) remove unused code (#27446) 2023-11-23 11:53:06 +08:00
b79f5d77f1 [improve](move-memtable) improve logging messages (#27443) 2023-11-23 11:46:29 +08:00
33de92cc61 [improve](nereids) support agg function of count(const value) pushdown #26677
support sql: select count(1)-count(not null) from table, the agg of count could push down.
2023-11-23 11:26:06 +08:00
Pxl
301bfe4d5d [Bug](mark-join) fix mark join report error when probe block have column do not output (#27360)
fix mark join report error when probe block have column do not output
2023-11-23 11:16:02 +08:00
42c32c584b [case](regression) test invalid jsonpaths (#27359)
Co-authored-by: qinhao <qinhao@newland.com.cn>
2023-11-23 10:16:34 +08:00
5b8aaf96d2 [fix](planner)scan node should project all required expr from parent node (#26886) 2023-11-23 09:44:21 +08:00
179d4ab754 [bugfix](potential bug) add lock when modify total tasks number (#27428)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-11-23 09:28:09 +08:00
fe5c4e3b46 [pipelineX](api) Fix core dump for pipelineX API (#27437) 2023-11-23 00:24:32 +08:00
044a295541 [performance](Planner): optimize getStringValue() in DateLiteral (#27363)
- reduce cost of `getStringValue()` 
- original code don't consider `microsecond` part in `getStringValue()`
2023-11-22 22:42:44 +08:00
1b3512d942 [pipelineX](bug) Fix cancel timeout (#27396) 2023-11-22 22:31:34 +08:00
a68b62b421 [refactor](datatype) Introduce a toolkits for clarify datelike types usage (#26810) 2023-11-22 22:08:13 +08:00
19c36dcc86 [Fix](statistics)Fix auto job start time incorrect bug (#27402)
Before, the auto analyze job start time was the job creation time, not the start to execute time, which is inaccurate. This pr is to change the start time to the first task start to execute time.
2023-11-22 21:38:08 +08:00
3e1a5b6e29 [profile](rf) make runtime filter profile readable (#27323) 2023-11-22 21:13:35 +08:00
0302a9d026 [fix](fe) slots in having clause should be set to need materialized (#27412) 2023-11-22 19:47:09 +08:00
7cdad51133 [feature](invert index) add the reader buffer as a parameter (#27339)
The inverted index read buffer is configurable
2023-11-22 19:29:20 +08:00
f49722163e [pipelineX](bug) Fix query timeout due to broadcast (#27398) 2023-11-22 19:25:04 +08:00
a2a6a722eb [test](regression) add routine load command test (#27384) 2023-11-22 18:55:35 +08:00
96e3636905 [Fix](Group_commit) Fix group commit regression test failure (#27380) 2023-11-22 18:55:20 +08:00
c7e3d74ffc [minor](stats) Report error with more friendly meesage when timeout (#27197) 2023-11-22 04:50:09 -06:00
cfb6af295f [fix](stats) Fix creating too many tasks on new env #27364
If there exists huge datasets with many database and may tables and many columns, Auto collector might be submit too many jobs which would occupy too much of FE memory.

In this PR, limit job each round could submit up to 5
2023-11-22 16:53:31 +08:00
6a48abeb80 [feature](Nereids) support queries tvf (#27138) 2023-11-22 02:47:16 -06:00
fd3c42d8cf [fix](test) order by clause in test_map (#27390) 2023-11-22 16:43:31 +08:00
732a3fa9c8 [fix](stats) fix auto collector always create sample job no matter the table size (#26968) 2023-11-22 02:42:40 -06:00
127525ebe2 [hotfix](jdbc catalog) fix realColumnNames serialize npe (#27280)
In the previous PR #27124, we used `objectMapper.readValue` for deserialization. However, this method does not handle null fields, which can lead to issues when upgrading from older versions. Specifically, if a required field is missing in the persistent data, `String realColumnNamesJson = serializeMap.get(REAL_COLUMNS);` will return null, resulting in deserialization errors and frontend startup failure. This issue is likely to occur when upgrading from an older version that uses Jdbc Catalog to a new version including PR #27124. As this represents a specific upgrade scenario involving compatibility with old version data structures, it was not covered in the regular PR test cases. Given the specificity and difficulty in replicating such a scenario, no special test cases were added for this PR.
2023-11-22 15:22:06 +08:00
39663119ca [minor](doc) Add sql manual for stats #27176 2023-11-22 13:15:42 +08:00
e06e976a8b [test](case) delete duplicate pipelineX cases (#27381) 2023-11-22 12:58:30 +08:00
5442e8d1fc [pipelineX](dependency) split different dependencies (#27366) 2023-11-22 12:50:39 +08:00
604842526b [improvement](expr) improve if expr performance (#27296) 2023-11-22 12:48:06 +08:00
1441387e5b [improvement](doris compose) improve start be container speed (#27368) 2023-11-22 11:04:04 +08:00
Pxl
b541de7a03 do not push down agg on aggregate column (#27356)
do not push down agg on aggregate column
2023-11-22 10:53:29 +08:00
b821672f8b [test](regression) add 'sync' for som stream load (#27357) 2023-11-22 10:52:34 +08:00
eaa1ca7143 [fix](fe) Fix show frontends npt in some situations (#27295)
```
java.lang.NullPointerException: null
    at com.sleepycat.je.rep.util.ReplicationGroupAdmin.getMasterSocket(ReplicationGroupAdmin.java:191)
    at com.sleepycat.je.rep.util.ReplicationGroupAdmin.doMessageExchange(ReplicationGroupAdmin.java:607)
    at com.sleepycat.je.rep.util.ReplicationGroupAdmin.getGroup(ReplicationGroupAdmin.java:406)
    at org.apache.doris.ha.BDBHA.getElectableNodes(BDBHA.java:132)
    at org.apache.doris.common.proc.FrontendsProcNode.getFrontendsInfo(FrontendsProcNode.java:84)
    at org.apache.doris.qe.ShowExecutor.handleShowFrontends(ShowExecutor.java:1923)
    at org.apache.doris.qe.ShowExecutor.execute(ShowExecutor.java:355)
    at org.apache.doris.qe.StmtExecutor.handleShow(StmtExecutor.java:2113)
    ...
```
2023-11-22 10:46:59 +08:00
c332811721 [fix](ci) fix bug of new trigger (#27379)
* [fix](ci) fix bug of new trigger
Co-authored-by: stephen <hello-stephen@qq.com>
2023-11-22 10:35:39 +08:00
e7a2b7b862 [comm](github) add submodule create-or-update-comment (#27301) 2023-11-22 10:24:38 +08:00
4fbcad9c7c [minor](show_export) make result of file url usable (#27209)
* [minor](show_export) make result of file url usable

* update regression-test

* update regression-test
2023-11-22 10:14:45 +08:00
34f500ac2a [github](action) optimize trigger teamcity pipeline (#27257)
optimize triiger teamcity pipeline
2023-11-22 09:34:47 +08:00
9b59bc14b5 [test](Export) add show export regression testes (#27140) 2023-11-22 00:13:30 +08:00
6e86bf5b1b [test](decimalv2) add some regression cases about decimalv2 (#27352)
All cases' results are tested and passed with decimalv3
Cases about:
Calculation ( +, - , *, /)
Kinds of predicates(<, >, =, <>, in, not in, is null, is not null)
Load test(from csv and select into)
Runtime filter
Delete conditions
Key columns(agg/duplicate/uniq model, distributed/partition, bitmap index...)
2023-11-21 21:36:20 +08:00
c9b959d2d8 [opt](Nereids) AssertNumRows node should triger runtime filter pruning #27279
1. optimize rf prune when col stats are not avaliable
2. add regression case to check plan and rf for tpcds_sf100 with stats
3. add regression case to check plan and rf for tpcds_sf100 without stats
2023-11-21 21:00:41 +08:00
1cd1c58eee [Feature](group commit) move group_commit_interval_ms from be.conf to table property (#27116) 2023-11-21 20:50:02 +08:00
c1435c0589 [regression test](routine test) add case for send_batch_parallelism (#27333) 2023-11-21 20:43:20 +08:00
a0fab03289 [case](regression-test) Add backup & restore regression test with auto Bucket and dynamic partition (#27033)
* (Case)Add backup & restore regression test
1. (case)Table with auto Bucket
2. Table with dynamic partition
(case)set "reserve_dynamic_partition_enable = true" on restore properties
(case)set "reserve_dynamic_partition_enable = false" on restore properties

Co-authored-by: Bears0haunt <bearshaunt0@gamil.com>
2023-11-21 20:32:45 +08:00
2f04873da9 [fix](clone) Fix engine_clone file exist (#27361)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-11-21 20:30:27 +08:00
d541424936 do not check invisible column stats (#27201)
1. forbid_unknown_col_stats check ignore invisible columns
2. a better error message when meet unknown col stats
2023-11-21 19:46:16 +08:00
ffd4face00 [refactor](callonce) remove callonce usage in pipeline task to get full stack trace when core (#27331)
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-11-21 19:42:39 +08:00
dea40e7095 [fix](Nereids): NullSafeEqual should be in HashJoinCondition (#27127)
Originally, we just put `EqualTo` in `HashJoinCondition`, we also need to allow `NullSafeEqual`
2023-11-21 19:08:14 +08:00