Commit Graph

6504 Commits

Author SHA1 Message Date
db317841a0 [hotfix](editlog) Fix upsert replay on follower not contains loadedTableIndexIds (#26597)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-11-09 12:21:43 +08:00
5d52162484 [Test](statistics) Add test cases for external table statistics (#26511)
1. Test for close and open auto collection for external catalog.
2. Test for analyze table table_name (column) and whole table.
2023-11-09 12:12:29 +08:00
7df60a4980 [Refactor](Tvf) delete some unused code of tvf and add doc for queries tvf (#26460)
1. delete some unused code of tvf
2. add doc for `queries` tvf: #25051
2023-11-09 09:06:09 +08:00
b3ae7f04f9 [fix](backup) Add repo id to local meta/info files to avoid overwriting (#26536)
The local meta/info files generated during backup are not distinguished
by repo names. If two backup jobs with the same name are submitted to
different repos at the same time, meta/info may be overwritten by another
backup job.
2023-11-08 22:28:49 +08:00
8e332fa979 (selectdb-cloud) Reduce FE db lock range for ShowDataStmt (#26588)
Reduce read lock critical sections and avoid execution timeouts
2023-11-08 22:24:17 +08:00
e92d2fcb5a [improvement](group commit) Group commit insert into can be executed on observer fe (#26589) 2023-11-08 22:10:06 +08:00
5bcf6bfd46 [fix](jdbc catalog) fix mysql zero date (#26569) 2023-11-08 21:41:56 +08:00
06343e6d68 [opt](nereids)replace scan by empty relation when all partitions are pruned (#26514)
* replace scan by empty relation when all partitions are pruned
2023-11-08 20:54:35 +08:00
e718952e89 [fix](nereids)only enable colocate scan for one phase global parttion topn in some condition (#26473) 2023-11-08 20:46:40 +08:00
0c1458f21f [fix](planner)isnull predicate can't be safely constant folded in inlineview (#25377)
disable is null predicate constant fold rule for inline view
consider sql
select c.*
from (
select a.*, b.x
from test_insert a left join
(select 'some_const_str' x from test_insert) b on true
) c
where c.x is null;

when push “c.x is null” into c, after folding constant rule, it will get empty result. Because x is 'some_const_str' and "x is null" will be evaluated to false. This is wrong.
2023-11-08 20:46:29 +08:00
d749d99fe2 [fix](nereids)don't normalize column name for base index (#26476) 2023-11-08 20:45:58 +08:00
223be6947c [opt](Nereids) let DataType toSql same with legacy planner (#26576) 2023-11-08 05:34:32 -06:00
45c2fa62a4 [pipeline](exec) disable shared scan in default and disable shared scan in limit with where scan (#25952) 2023-11-08 17:51:12 +08:00
a6d2013802 [opt](nereids) use 2 phase agg above union all (#26245)
forbid one phase agg for pattern: agg-unionAll
one phase agg plan: agg-union-hashDistribute-children
two phase agg plan: agg(global) - hashDistribute-agg(local)-union-randomDistribute
the key point is the cost of randomDistribute is much lower than the hashDistribute, and hence two-phase agg wins.
2023-11-08 17:15:53 +08:00
96d2e3394a [opt](meta) Improve the performance of getting expr name (#26341)
CaseFormat.UPPER_CAMEL.to(CaseFormat.LOWER_UNDERSCORE, name)
It's time-consuming when call many times. So lazy call when necessary
2023-11-08 03:14:15 -06:00
58bf79f79e [fix](move-memtable) pass load stream num to backends (#26198) 2023-11-08 16:16:33 +08:00
a3666aa87e [feature](decimal) support decimal256 when creating table (#26308) 2023-11-08 15:21:01 +08:00
f80495da83 [fix](Nereids) ban right outer, right anti, full outer with bucket shuffle (#26529)
if left bucket has no data, we do not generate left bucket instance.
These join should reserve all right side data. But because left instance
is not exists. So right data will be discard since no dest be set.

We ban these join temporarily until we could generate all instance
for left side in Coordinator.
2023-11-08 01:16:50 -06:00
fc304c0e7c (metric) add histogramJsonMetric and nodeInfo (#26172)
Add histogramJsonMetric and nodeInfo to the interface "http://fe_host:http_port/metrics?type=json".
2023-11-08 14:46:18 +08:00
7bad2e1d9f [opt](nereids) infer result column name in ctas and query stmt (#26055)
Infer name if it is an expression and doesn't alias artificially when create or select stmt in nereids.
The infer name strategy is the same as #24990
2023-11-07 21:28:48 -06:00
47ba4aaf30 [Enhancement](load) add timer and partitions number limit (#26549)
add timer and partitions number limit
2023-11-08 11:22:40 +08:00
c93c8f6105 [opt](nereids) make AGG_SCALAR_SUBQUERY_TO_WINDOW_FUNCTION rewrite rule #25969 2023-11-08 11:04:08 +08:00
290070074a [refactor](stats) refactor collection logic and opt some config (#26163)
1. not collect partition stats anymore
2. merge insert of stats
3. delete period collector since it is useless
4. remove enable_auto_sample
5. move some config related to stats to global session variable

Before this PR, when analyze  a table, the insert count equals column count times 2

After this PR, insert count of analyze table would reduce to column count / insert_merge_item_count.

According to my test, when analyzing  tpch lineitem, the insert sql count is 1
2023-11-08 11:03:44 +08:00
1544110c1b [feature-wip](arrow-flight)(step4) Support other DML and DDL statements, besides Select (#25919)
Design Documentation Linked to #25514
2023-11-08 10:50:42 +08:00
806461721c [opt](Nereids) remove Nondeterministic trait from date related functions (#26444) 2023-11-07 20:43:37 -06:00
daea751a98 [Improvement](auditlog) add column catalog for audit log and audit log table (#26403) 2023-11-08 10:25:15 +08:00
Pxl
3cdbb6e637 [Bug](materialized-view) fix some bugs on create mv with percentile_approx (#26528)
1. percentile_approx have wrong symbol
2. fnCall.getParams() get obsolete childrens
2023-11-08 10:09:37 +08:00
3faf3b4118 [chore] Print FE version even if it has been started (#26427)
In the previous implementation, `bin/start_fe.sh --version` will
complain that "Frontend running as process xxx. Stop it first."

To show version
1. `bin/start_fe.sh --version` will print version info to fe.out
2. `bin/start_fe.sh --console --version` will print version info to stdout
2023-11-07 22:33:02 +08:00
ceccc451fa [enhancement](Nereids): add LOG info to show the phase of NereidsPlanner. (#26538)
Add LOG info to show the phase of NereidsPlanner, we can use these info to debug.
2023-11-07 21:46:54 +08:00
2be6c9ff7d [enhancement](Nereids): when the DPhyper failed, roll back to cascades without join reorder (#26390)
when the DPhyper failed, roll back to cascades without join reorder
2023-11-07 20:05:40 +08:00
5e9a23e643 [fix](prepare statement) Not supported such prepared statement if prepare a forward master sql (#26512) 2023-11-07 19:41:44 +08:00
ad1f635070 [Feature](auditloader) Plugin auditloader use auth token to avoid using cleartext passwords in config (#26278)
Doris FE will check if stream load http request has auth token after checking password failed;
Plugin audit-log loader can use auth token if plugin config set use_auth_token to true

Co-authored-by: Mingyu Chen <morningman.cmy@gmail.com>
2023-11-07 19:14:57 +08:00
b0788652bd [bugfix](clickhouse) fix datetime convert error. (#26128) 2023-11-07 17:16:07 +08:00
efd1aa3016 [Revert](code-style) revert FE code-format #25033 and #26488 (#26505) 2023-11-07 16:37:24 +08:00
f138aaa07a [fix](nereids) unnest in-subquery with agg node in proper condition (#25800)
consider sql having in-subquery

SELECT count(*)
        FROM sub_query_correlated_subquery6
        WHERE k1 IN 
            (SELECT k1
            FROM 
                (**SELECT k1,
                sum(k3) AS bbb,
                count(k2) AS aaa
                FROM sub_query_correlated_subquery7
                WHERE k1 > 0
                        AND k3 > 0
                GROUP BY  k1** ) y
                WHERE y.aaa>0
                        AND k1>1); 

The subquery part having agg is un-correlated, which can be unnested.

on the other side:
SELECT count(*)
                    FROM sub_query_correlated_subquery6
                    WHERE k1 IN 
                        (SELECT k1
                        FROM 
                            (**SELECT k1,
                            sum(k3) AS bbb,
                            count(k2) AS aaa
                            FROM sub_query_correlated_subquery7
                            WHERE k1 > 0
                                    AND k3 > 0 and sub_query_correlated_subquery6.k1 > 2
                            GROUP BY  k1** ) y
                            WHERE y.aaa>0
                                    AND k1>1);

The subquery part having agg is correlated, which can't be unnested.
2023-11-06 20:35:13 -06:00
65304ba216 [fix](code-style) Adapt to checkstyle and spotless (#26488) 2023-11-07 00:23:39 +08:00
6983736cce [Fix](partial update) Fix core when successfully schema change and load during a partial update (#26210) 2023-11-06 23:16:05 +08:00
e42cb88c2d [refactor](profile) remove unused code in profile (#26404) 2023-11-06 18:13:37 +08:00
0057ec36a3 [feature](profilev2) replace active time to exectime (#26310) 2023-11-06 18:12:39 +08:00
85a1db4b6c [fix](nereids)select base index if select mv fails (#25715) 2023-11-06 17:57:19 +08:00
64ea450ac7 [enhancement](Nereids): add Unit Test for PushdownTopNThroughJoin (#26447) 2023-11-06 16:54:36 +08:00
1a83a39aec Revert "[fix](auto-partition) Fix auto partition concurrent conflict (#26166)" (#26448)
This reverts commit f22611769944e78c28f1b0a1eeb7b7414a16e8db.
2023-11-06 16:39:19 +08:00
d088cba2b1 [feature](code-style)add spotless plugin (#25033) 2023-11-06 14:01:39 +08:00
2cc68381ec [feature](binlog) Add ingest_binlog/http_get_snapshot limit download speed && Add async ingest_binlog (#26323) 2023-11-06 11:14:44 +08:00
c0ed5f74ad [fix](Nereids) storage later agg rule process agg children by mistake (#26101)
update Project#findProject
agg function's children could be any expression rather than only slot.
we use Project#findProject to process them. But this util could only
process slot. This PR update this util to let it could process all type
expression.
2023-11-05 21:07:52 -06:00
f226117699 [fix](auto-partition) Fix auto partition concurrent conflict (#26166) 2023-11-06 10:34:26 +08:00
d9bd1fad67 [fix](spark-load)fix-Unique-key-with-MOR-by-sparkload (#26383)
When a Unique key table carries the `enable_unique_key_merge_on_write` attribute, the value of the agg type is none. Therefore, when doing sparkload, we need to specify the agg type as `REPLACE`.
2023-11-06 09:56:46 +08:00
6ef2e62600 [FIX](struct) fix struct be nested when read will make core (#26270) 2023-11-03 09:55:39 -05:00
11059ca4ed [fix](nereids)SimplifyRange rule may mess up and/or predicate (#26304) 2023-11-03 20:38:07 +08:00
646348ccc4 [fix](nereids)add visitMarkJoinReference method in ExpressionDeepCopier (#25874) 2023-11-03 20:34:29 +08:00