Commit Graph

3042 Commits

Author SHA1 Message Date
124a8a9b34 [enhancement](regression) add profile before datev2 insert for debug (#26617)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2023-11-09 12:21:15 +08:00
5d52162484 [Test](statistics) Add test cases for external table statistics (#26511)
1. Test for close and open auto collection for external catalog.
2. Test for analyze table table_name (column) and whole table.
2023-11-09 12:12:29 +08:00
57ed781bb6 [fix](regression-test) Add tvf regression tests (#26455) 2023-11-09 12:09:32 +08:00
d1438a8563 [Fix](orc-reader) Fix orc complex types when late materialization was turned on by disabling late materialization in this case. (#26548)
Fix orc complex types when late materialization was turned on in orc reader by disabling late materialization in this case.
2023-11-09 12:05:43 +08:00
f6b7046a6e [fix](regression-test) add tests for jdbc catalog (#26608) 2023-11-09 11:59:35 +08:00
95f74f1544 [FIX](complextype)fix shrink in topN for complex type #26609 2023-11-09 10:56:14 +08:00
55b2988bfd [Opt](date_add/sub) Throw exception when result of date_add/sub out of range (#26475) 2023-11-09 08:46:51 +08:00
faaf0ecc85 [regression-test](framework) Support running tests multiple times and reporting correctly to TeamCity (#26606) 2023-11-08 09:42:53 -06:00
9c828ff79c [cases](regression-test) Add backup & restore test case of dup table (#26490)
Co-authored-by: Bears0haunt <bearshaunt0@gamil.com>
2023-11-08 22:30:01 +08:00
18b3d0ec6b [cases](regression-test) add unique and duplicate backup and restore … (#26491)
* [cases](regression-test) add unique and duplicate backup and restore table models

* Add delete and mor scenes
2023-11-08 22:29:34 +08:00
b3ae7f04f9 [fix](backup) Add repo id to local meta/info files to avoid overwriting (#26536)
The local meta/info files generated during backup are not distinguished
by repo names. If two backup jobs with the same name are submitted to
different repos at the same time, meta/info may be overwritten by another
backup job.
2023-11-08 22:28:49 +08:00
ee6e6911da [regression-test](stream load) Invalid merge type check (#26599) 2023-11-08 22:28:30 +08:00
b7a2c2e9c4 [chore](regression) Do stale resource reclaim before executing cold heat separation p2 case(#26596) 2023-11-08 22:14:54 +08:00
5bcf6bfd46 [fix](jdbc catalog) fix mysql zero date (#26569) 2023-11-08 21:41:56 +08:00
06343e6d68 [opt](nereids)replace scan by empty relation when all partitions are pruned (#26514)
* replace scan by empty relation when all partitions are pruned
2023-11-08 20:54:35 +08:00
0c1458f21f [fix](planner)isnull predicate can't be safely constant folded in inlineview (#25377)
disable is null predicate constant fold rule for inline view
consider sql
select c.*
from (
select a.*, b.x
from test_insert a left join
(select 'some_const_str' x from test_insert) b on true
) c
where c.x is null;

when push “c.x is null” into c, after folding constant rule, it will get empty result. Because x is 'some_const_str' and "x is null" will be evaluated to false. This is wrong.
2023-11-08 20:46:29 +08:00
d749d99fe2 [fix](nereids)don't normalize column name for base index (#26476) 2023-11-08 20:45:58 +08:00
223be6947c [opt](Nereids) let DataType toSql same with legacy planner (#26576) 2023-11-08 05:34:32 -06:00
ec87401581 Fix workload group regression test failed (#26579) 2023-11-08 19:23:49 +08:00
a6d2013802 [opt](nereids) use 2 phase agg above union all (#26245)
forbid one phase agg for pattern: agg-unionAll
one phase agg plan: agg-union-hashDistribute-children
two phase agg plan: agg(global) - hashDistribute-agg(local)-union-randomDistribute
the key point is the cost of randomDistribute is much lower than the hashDistribute, and hence two-phase agg wins.
2023-11-08 17:15:53 +08:00
f018b00646 [ci](perf) add new pipeline of tpch-sf100 (#26334)
* [ci](perf) add new pipeline of tpch-sf100
Co-authored-by: stephen <hello-stephen@qq.com>
2023-11-08 15:32:02 +08:00
a3666aa87e [feature](decimal) support decimal256 when creating table (#26308) 2023-11-08 15:21:01 +08:00
f80495da83 [fix](Nereids) ban right outer, right anti, full outer with bucket shuffle (#26529)
if left bucket has no data, we do not generate left bucket instance.
These join should reserve all right side data. But because left instance
is not exists. So right data will be discard since no dest be set.

We ban these join temporarily until we could generate all instance
for left side in Coordinator.
2023-11-08 01:16:50 -06:00
5d4557938a [regression-test](fix) fix export_struct bug (#26561) 2023-11-08 14:57:07 +08:00
0f3e97f9c5 [regression-test][framework] support cases that can only run in non-concurrent-mode. (#26487) 2023-11-08 12:46:36 +08:00
7bad2e1d9f [opt](nereids) infer result column name in ctas and query stmt (#26055)
Infer name if it is an expression and doesn't alias artificially when create or select stmt in nereids.
The infer name strategy is the same as #24990
2023-11-07 21:28:48 -06:00
290070074a [refactor](stats) refactor collection logic and opt some config (#26163)
1. not collect partition stats anymore
2. merge insert of stats
3. delete period collector since it is useless
4. remove enable_auto_sample
5. move some config related to stats to global session variable

Before this PR, when analyze  a table, the insert count equals column count times 2

After this PR, insert count of analyze table would reduce to column count / insert_merge_item_count.

According to my test, when analyzing  tpch lineitem, the insert sql count is 1
2023-11-08 11:03:44 +08:00
806461721c [opt](Nereids) remove Nondeterministic trait from date related functions (#26444) 2023-11-07 20:43:37 -06:00
Pxl
3cdbb6e637 [Bug](materialized-view) fix some bugs on create mv with percentile_approx (#26528)
1. percentile_approx have wrong symbol
2. fnCall.getParams() get obsolete childrens
2023-11-08 10:09:37 +08:00
607a5d25f1 [feature](streamload) support HTTP request with chunked transfer (#26520) 2023-11-08 10:07:05 +08:00
70bc8600a9 [fix](regression) fix regression framework bug: if real test result is negative, it will miss check test result (#25734) 2023-11-08 09:05:58 +08:00
ef95e962c7 [fix](timev2) fix Type not implemented in fold by be (#26478) 2023-11-07 17:25:20 +08:00
f0bf3fadad [test](executor)Add workload group regression test (#26446) 2023-11-07 16:37:54 +08:00
9687932d57 [refactor](function) improve compoundPred optimization work with children is nullable (#26160)
before this optimization work has limit, it's children must not nullable.
2023-11-07 13:52:10 +08:00
f138aaa07a [fix](nereids) unnest in-subquery with agg node in proper condition (#25800)
consider sql having in-subquery

SELECT count(*)
        FROM sub_query_correlated_subquery6
        WHERE k1 IN 
            (SELECT k1
            FROM 
                (**SELECT k1,
                sum(k3) AS bbb,
                count(k2) AS aaa
                FROM sub_query_correlated_subquery7
                WHERE k1 > 0
                        AND k3 > 0
                GROUP BY  k1** ) y
                WHERE y.aaa>0
                        AND k1>1); 

The subquery part having agg is un-correlated, which can be unnested.

on the other side:
SELECT count(*)
                    FROM sub_query_correlated_subquery6
                    WHERE k1 IN 
                        (SELECT k1
                        FROM 
                            (**SELECT k1,
                            sum(k3) AS bbb,
                            count(k2) AS aaa
                            FROM sub_query_correlated_subquery7
                            WHERE k1 > 0
                                    AND k3 > 0 and sub_query_correlated_subquery6.k1 > 2
                            GROUP BY  k1** ) y
                            WHERE y.aaa>0
                                    AND k1>1);

The subquery part having agg is correlated, which can't be unnested.
2023-11-06 20:35:13 -06:00
daeca7d414 [feature](doris compose) support multiple disk types and other improvement (#25924) 2023-11-06 23:23:09 +08:00
6983736cce [Fix](partial update) Fix core when successfully schema change and load during a partial update (#26210) 2023-11-06 23:16:05 +08:00
3e8d2f6986 [fix](regression-test) Fix regiressin test syncer suit use master fe directly (#26456)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-11-06 22:51:54 +08:00
eeb8704b66 [fix](regression-test) add more check to address flaky test_partial_update_with_delete_stmt (#26474) 2023-11-06 22:45:13 +08:00
2ba2cd22b0 [fix](regression) Fix unstable regression case introduced by #26384 (#26453) 2023-11-06 18:34:14 +08:00
2cc68381ec [feature](binlog) Add ingest_binlog/http_get_snapshot limit download speed && Add async ingest_binlog (#26323) 2023-11-06 11:14:44 +08:00
f831774121 [test](regression) Add more regression test for FE (#26384) 2023-11-06 11:10:37 +08:00
c0ed5f74ad [fix](Nereids) storage later agg rule process agg children by mistake (#26101)
update Project#findProject
agg function's children could be any expression rather than only slot.
we use Project#findProject to process them. But this util could only
process slot. This PR update this util to let it could process all type
expression.
2023-11-05 21:07:52 -06:00
a5b9d61f43 [bug](udf) fix java-udf function const column without arguments (#26409) 2023-11-06 10:34:09 +08:00
6ef2e62600 [FIX](struct) fix struct be nested when read will make core (#26270) 2023-11-03 09:55:39 -05:00
646348ccc4 [fix](nereids)add visitMarkJoinReference method in ExpressionDeepCopier (#25874) 2023-11-03 20:34:29 +08:00
8c3e173553 [fix](nereids)fix bug of select mv in nereids (#26235)
* revert some change from pr26192
* disable some case for nereids
2023-11-03 20:28:55 +08:00
d367d9b585 [fix](show_proc) fix show statistic proc dir to ensure that result only contains dbs in internal catalog (#26254) 2023-11-03 20:12:06 +08:00
7730a9025e [Fix](Regression-test) add test for tvf (#26322) 2023-11-03 19:07:07 +08:00
fd92810df8 [FIX](regresstest)fixed out file for test-cast-map-function test (#26389) 2023-11-03 05:47:10 -05:00