Commit Graph

14606 Commits

Author SHA1 Message Date
70bc8600a9 [fix](regression) fix regression framework bug: if real test result is negative, it will miss check test result (#25734) 2023-11-08 09:05:58 +08:00
a6756b4660 [pipelineX](bug) Fix broadcast buffer reference count (#26545) 2023-11-08 00:14:48 +08:00
4995ca8fba [fix](move-memtable) ensure segment is flushed before add segment (#26522) 2023-11-07 22:42:16 +08:00
32b36d3c9c [refactor](move-memtable) rename proto OpenStreamSink to OpenLoadStream (#26527) 2023-11-07 22:41:20 +08:00
3faf3b4118 [chore] Print FE version even if it has been started (#26427)
In the previous implementation, `bin/start_fe.sh --version` will
complain that "Frontend running as process xxx. Stop it first."

To show version
1. `bin/start_fe.sh --version` will print version info to fe.out
2. `bin/start_fe.sh --console --version` will print version info to stdout
2023-11-07 22:33:02 +08:00
5d80e7dc2f [Improvement](pipelineX) Improve local exchange on pipelineX engine (#26464) 2023-11-07 22:11:44 +08:00
ceccc451fa [enhancement](Nereids): add LOG info to show the phase of NereidsPlanner. (#26538)
Add LOG info to show the phase of NereidsPlanner, we can use these info to debug.
2023-11-07 21:46:54 +08:00
2be6c9ff7d [enhancement](Nereids): when the DPhyper failed, roll back to cascades without join reorder (#26390)
when the DPhyper failed, roll back to cascades without join reorder
2023-11-07 20:05:40 +08:00
5e9a23e643 [fix](prepare statement) Not supported such prepared statement if prepare a forward master sql (#26512) 2023-11-07 19:41:44 +08:00
2bb3ef1981 [refactor](scan) delete bloom_filter_predicate (#26499) 2023-11-07 19:37:31 +08:00
d6eb3324a1 [cleanup](load) remove unused code in sink v2 header (#26521) 2023-11-07 19:35:12 +08:00
ad1f635070 [Feature](auditloader) Plugin auditloader use auth token to avoid using cleartext passwords in config (#26278)
Doris FE will check if stream load http request has auth token after checking password failed;
Plugin audit-log loader can use auth token if plugin config set use_auth_token to true

Co-authored-by: Mingyu Chen <morningman.cmy@gmail.com>
2023-11-07 19:14:57 +08:00
38a14c3325 [docs](fix) add bitmap_remove in sidebars.json (#26523) 2023-11-07 19:01:27 +08:00
2feed57f47 [Fix](fs_benchmark_tools) Fix run_fs_benchmark.sh classpath issue. (#26183)
Fix run_fs_benchmark.sh classpath issue.
2023-11-07 18:43:30 +08:00
a404ff5ab9 [fix](regression) fix group commit regression test (#26519) 2023-11-07 18:17:45 +08:00
ef95e962c7 [fix](timev2) fix Type not implemented in fold by be (#26478) 2023-11-07 17:25:20 +08:00
b0788652bd [bugfix](clickhouse) fix datetime convert error. (#26128) 2023-11-07 17:16:07 +08:00
3ad8e27b09 [Fix](autoinc) Init auto increment info in VOlapTableSinkV2 (#26502) 2023-11-07 16:51:38 +08:00
f0bf3fadad [test](executor)Add workload group regression test (#26446) 2023-11-07 16:37:54 +08:00
efd1aa3016 [Revert](code-style) revert FE code-format #25033 and #26488 (#26505) 2023-11-07 16:37:24 +08:00
8da1a9a370 [pipeline](fix) remove unreasonable CHECK (#26504) 2023-11-07 15:48:07 +08:00
277329c035 [fix](auditlog) fix without lock in QueryStatisticsRecvr find #26440 2023-11-07 13:53:22 +08:00
9687932d57 [refactor](function) improve compoundPred optimization work with children is nullable (#26160)
before this optimization work has limit, it's children must not nullable.
2023-11-07 13:52:10 +08:00
f138aaa07a [fix](nereids) unnest in-subquery with agg node in proper condition (#25800)
consider sql having in-subquery

SELECT count(*)
        FROM sub_query_correlated_subquery6
        WHERE k1 IN 
            (SELECT k1
            FROM 
                (**SELECT k1,
                sum(k3) AS bbb,
                count(k2) AS aaa
                FROM sub_query_correlated_subquery7
                WHERE k1 > 0
                        AND k3 > 0
                GROUP BY  k1** ) y
                WHERE y.aaa>0
                        AND k1>1); 

The subquery part having agg is un-correlated, which can be unnested.

on the other side:
SELECT count(*)
                    FROM sub_query_correlated_subquery6
                    WHERE k1 IN 
                        (SELECT k1
                        FROM 
                            (**SELECT k1,
                            sum(k3) AS bbb,
                            count(k2) AS aaa
                            FROM sub_query_correlated_subquery7
                            WHERE k1 > 0
                                    AND k3 > 0 and sub_query_correlated_subquery6.k1 > 2
                            GROUP BY  k1** ) y
                            WHERE y.aaa>0
                                    AND k1>1);

The subquery part having agg is correlated, which can't be unnested.
2023-11-06 20:35:13 -06:00
16644eff7f [opt](load) optimize the performance of row distribution (#25546)
For non-pipeline non-sinkv2:
before: 14s
now: 6s-
For pipeline + sinkv2:
before: 230ms *48 instances
now: 38ms *48 instances
2023-11-07 10:04:59 +08:00
fa7a38b587 [fix](runtime filter) append late arrival runtime filters in vfilecanner (#25996)
`VFileScanner` will try to append late arrival runtime filters in each loop of `ScannerScheduler::_scanner_scan`.  However, `VFileScanner::_get_next_reader` only generates the `_push_down_conjuncts` in the first loop, so the late arrival runtime filters are ignored.
2023-11-07 09:50:35 +08:00
65304ba216 [fix](code-style) Adapt to checkstyle and spotless (#26488) 2023-11-07 00:23:39 +08:00
84b90abeeb [typo](docs)Improve stream load document (#25893) 2023-11-06 23:23:45 +08:00
daeca7d414 [feature](doris compose) support multiple disk types and other improvement (#25924) 2023-11-06 23:23:09 +08:00
6983736cce [Fix](partial update) Fix core when successfully schema change and load during a partial update (#26210) 2023-11-06 23:16:05 +08:00
b99afcc7b5 [docs](x-load) Add table property min_load_replica_num doc (#26333) 2023-11-06 23:05:35 +08:00
3e8d2f6986 [fix](regression-test) Fix regiressin test syncer suit use master fe directly (#26456)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-11-06 22:51:54 +08:00
bf8793de70 [fix](load) fix merged row number miscounting because of race condition (#26470)
row numbers miscounting because of race condition, will cause load to
fail sometimes with warning 'the rows number written doesn't match'.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2023-11-06 22:50:32 +08:00
eeb8704b66 [fix](regression-test) add more check to address flaky test_partial_update_with_delete_stmt (#26474) 2023-11-06 22:45:13 +08:00
c9d90b636e [feature](pipelineX) add time unit when slow_dependency need to log (#26466) 2023-11-06 19:58:07 +08:00
2ba2cd22b0 [fix](regression) Fix unstable regression case introduced by #26384 (#26453) 2023-11-06 18:34:14 +08:00
e42cb88c2d [refactor](profile) remove unused code in profile (#26404) 2023-11-06 18:13:37 +08:00
0057ec36a3 [feature](profilev2) replace active time to exectime (#26310) 2023-11-06 18:12:39 +08:00
85a1db4b6c [fix](nereids)select base index if select mv fails (#25715) 2023-11-06 17:57:19 +08:00
bd89028306 [bug](pipelineX) Fix potential bug using broadcast shuffle (#26458) 2023-11-06 17:33:20 +08:00
64ea450ac7 [enhancement](Nereids): add Unit Test for PushdownTopNThroughJoin (#26447) 2023-11-06 16:54:36 +08:00
1a83a39aec Revert "[fix](auto-partition) Fix auto partition concurrent conflict (#26166)" (#26448)
This reverts commit f22611769944e78c28f1b0a1eeb7b7414a16e8db.
2023-11-06 16:39:19 +08:00
c0d9a8d53c [minor](pipelineX) refine error message for broadcast shuffle buffer (#26442) 2023-11-06 15:10:13 +08:00
d088cba2b1 [feature](code-style)add spotless plugin (#25033) 2023-11-06 14:01:39 +08:00
2cc68381ec [feature](binlog) Add ingest_binlog/http_get_snapshot limit download speed && Add async ingest_binlog (#26323) 2023-11-06 11:14:44 +08:00
f831774121 [test](regression) Add more regression test for FE (#26384) 2023-11-06 11:10:37 +08:00
c0ed5f74ad [fix](Nereids) storage later agg rule process agg children by mistake (#26101)
update Project#findProject
agg function's children could be any expression rather than only slot.
we use Project#findProject to process them. But this util could only
process slot. This PR update this util to let it could process all type
expression.
2023-11-05 21:07:52 -06:00
1e2a614a46 [fix](workflow) Fix failure test cases in BE UT (macOS) (#26425)
1. Fix memory issues in LoadStreamMgrTest.
2. Skip S3FileWriterTest by default because it depends on the environment in teamcity.
3. Fix VTimestampFunctionsTest.convert_tz_test.
2023-11-06 10:44:44 +08:00
f226117699 [fix](auto-partition) Fix auto partition concurrent conflict (#26166) 2023-11-06 10:34:26 +08:00
a5b9d61f43 [bug](udf) fix java-udf function const column without arguments (#26409) 2023-11-06 10:34:09 +08:00