Commit Graph

13444 Commits

Author SHA1 Message Date
f57c75f935 [doc](agg_state) add agg_state in data-model (#24348)
add agg_state in data-model
2023-09-15 11:45:21 +08:00
Pxl
15c8ff115c [Chore](checks) run sonar-cloud-java when fe changed (#24420)
run sonar-cloud-java when fe changed
2023-09-15 11:41:35 +08:00
df5b5ae0cb [improvement](create tablet) backend create tablet round robin among disks (#23218)
Backend choose disk by disk available bytes and tablet num. If both are equal, round robin among them.
2023-09-15 11:39:43 +08:00
2976e9d8e5 [imrovement](iterators) reduce memory for union iterators (#24197) 2023-09-15 11:35:56 +08:00
c5e7f55b63 [performance](executor) optimize time_round function (#23058)
optimize time_round function
2023-09-15 10:49:22 +08:00
Pxl
90e434a5dd [Chore](check) change SonarClout Analysis scheduled on push (#24410)
change SonarClout Analysis scheduled on push
2023-09-15 10:42:00 +08:00
dbd7733e02 [feature](regression) Add p2 level test for schema change (#20243) 2023-09-15 10:39:07 +08:00
00bb32cfc0 [opt](nereids) enable two phase partition topn opt #23870
Enable two phase partition topn optimization, instead of original full sort at the second phase.
E.g, partial plan of tpcds q67 is as following and a full sort after exchange will have performance impact, especially if the window column's ndv is very high and the number of window is huge.

------PhysicalTopN
--------filter((rk <= 100))
----------PhysicalWindow
------------PhysicalQuickSort
--------------PhysicalDistribute
----------------PhysicalPartitionTopN
------------------PhysicalProject

Under this scenario, the second phase full sort can be transformed to a global PhysicalPartitionTopN and reduce the cost from full sort. The plan will be optimized to the following:

------PhysicalTopN
--------filter((rk <= 100))
----------PhysicalWindow
------------PhysicalPartitionTopN
--------------PhysicalDistribute
----------------PhysicalPartitionTopN
------------------PhysicalProject
2023-09-15 10:30:34 +08:00
23f01ddf3a [feature](profile) support simply profile (#23377)
A Simplified Version of the Profile

Divided into three levels:
Level 2: The original profile.
Level 1: Instances with identical structures are merged, utilizing concatenation for info strings, and recording the extremum for time types.


Note that currently, this is purely experimental, simplifying the profile on the frontend (you can view profiles at any level).

Subsequently, we will transition the simplification process to the backend. At that point, due to the simplification being done on the backend, viewing profiles at other levels won't be possible.

Due to the issue with the pipeline structure, the active time does not accurately reflect the time of the operators.

```
set enable_simply_profile = false;
set enable_simply_profile = true;
```
2023-09-15 10:25:14 +08:00
320f1e9bbf [improve](routineload) improve show routine load output (#24264) 2023-09-15 10:22:47 +08:00
e0834b2f46 [chore](explain) add annotation in explain string whether nereids is ON #24394 2023-09-15 10:17:17 +08:00
14163df429 [github](config) forbid shell check in pytest directory #24378 2023-09-15 10:15:08 +08:00
9c681692bd Revert "[fix] fix http_stream retry mechanism (#23969)" (#24407)
This reverts commit 05e365ea137eb8c92b8e7eedc7d1435e83f065ae.
2023-09-15 10:07:53 +08:00
c5ef6cfea2 [fix](Table-Valued Function) fix be core when user sepcified empty column_separator using hdfs tvf (#24369) 2023-09-14 23:19:48 +08:00
5ba1f62da8 [enhancement](Nereids) make stats unchanged (#23737)
make stats unchanged when explore plan
2023-09-14 22:18:54 +08:00
66bd2a4862 [test](Nereids) add test push down filter (#24250)
Add test for pushDownFilterThroughProject
2023-09-14 22:13:41 +08:00
d4756d3118 [feature](Nereids): fold Cast(s as date/datetime) on FE (#24353)
cast("20210101" as Date) -> DateLiteral(2021, 1, 1)
2023-09-14 22:08:26 +08:00
f61e6483bf [enhancement](broker-load) support compress type for old broker load, and split compress type from file format (#23882) 2023-09-14 21:42:28 +08:00
0488c87a38 [typo][doc]modify broken link. (#23988) 2023-09-14 21:41:40 +08:00
05e365ea13 [fix] fix http_stream retry mechanism (#23969)
Co-authored-by: yiguolei <676222867@qq.com>
2023-09-14 21:41:11 +08:00
07720d3ff9 [feature](replica version) Add admin set replica version statement (#23706) 2023-09-14 21:12:00 +08:00
d20365cdcf [fix](transaction) fix publish txn fake succ (#24273) 2023-09-14 21:04:59 +08:00
0c30720c99 [Hotfix](merge-on-write) Fix when DeltaWriter don't report num_filtered_rows in TTabletCommitInfo (#24352) 2023-09-14 20:58:08 +08:00
321281da90 [Chore](Sonar)Set C++ code detection to scheduled scheduling (#24388)
Set C++ code detection to scheduled scheduling
2023-09-14 20:55:35 +08:00
927de33166 [config](log) disable StreamLoad log default and enable in regression pipeline (#24354)
disable StreamLoad log default and enable in regression pipeline
2023-09-14 20:47:26 +08:00
c6a92955ca [refacotr](optimizer) Remove useless check #24237
Check stats table status at first
Comment histgram_tbl check since it useless for now
Do preheat both in master and follower
2023-09-14 19:35:56 +08:00
6fe207eb4b [fix](broker) do not close filesystem(#24357)
same as #24128
To avoid Filesystem closed error
2023-09-14 18:36:09 +08:00
eb65cc6954 [Fix](nereids) eliminate_outer_join regression case fix #24262 2023-09-14 18:22:17 +08:00
Pxl
55de937651 [Chore](check) fix NoSuchFileException: /github/workspace/build_Release/compile_commands.json (#24380)
fix NoSuchFileException: /github/workspace/build_Release/compile_commands.json
2023-09-14 18:12:23 +08:00
3ee89aea35 [Feature](merge-on-write)Support ignore mode for merge-on-write unique table (#21773) 2023-09-14 18:03:51 +08:00
68b13ab50f [Fix](Full compaction) Fix full compaction by table id case (#24265) 2023-09-14 18:03:28 +08:00
Pxl
35c5d71549 [Improvement](join) some improvement of hash join (#23972)
some improvement of hash join
2023-09-14 17:55:35 +08:00
9c6734e68e [bugfix](index) Fix build index limitations (#24358)
1. skip existed index on column with different id on build index
2. allow build index for CANCELED or FINISHED state
2023-09-14 17:53:22 +08:00
eaa35649bc [fix](bdbje) handle ReplicaWriteException in BDBJEJournal.write (#24259)
* When BDBJEJournal.write meet `ReplicaWriteException`, we should not
  retry. Because at the monment the bdbje node state is `REPLICA` (not `MASTER`)
  if we still retry write, at the same time trigger election, the orgin `REPLICA`
  node may transfer to `MASTER` and will cause incorrect journalId

Co-authored-by: yiguolei <676222867@qq.com>
2023-09-14 17:49:28 +08:00
85e5b49d4c update structured streaming doc (#24016) 2023-09-14 17:47:27 +08:00
Pxl
3827549aba [Chore](checks) change SonarCloud Scan projectBaseDir to be to avoid include .java file (#24377)
change SonarCloud Scan projectBaseDir to be to avoid include .java file
2023-09-14 17:26:23 +08:00
d035a58374 [feature](nereids) support unnest subquery in LogicalOneRowRelation (#24355)
select (select 1);
before : 
ERROR 1105 (HY000): errCode = 2, detailMessage = Subquery is not supported in the select list.
after:
mysql> select (select 1);
+---------------------------------------------------------------------+
|  (SCALARSUBQUERY) (LogicalOneRowRelation ( projects=[1 AS `1`#0] )) |
+---------------------------------------------------------------------+
|                                                                   1 |
+---------------------------------------------------------------------+
1 row in set (0.61 sec)
2023-09-14 17:22:08 +08:00
0be0b8ff58 [opt](stats) Support display of auto analyze jobs (#24135)
### Support dispaly of auto analyze jobs

After this PR, users and DBA could use such grammar to check the execution status of auto analyze jobs:

```sql

SHOW AUTO ANALYZE [tbl_name] [WHERE STATE='SOME STATE']
```

Record count of history auto analyze job could be configured by setting FE option: auto_analyze_job_record_count, default value is 2000

### Enhance auto analyze

After this PR, auto jobs those created automatically will no longer execute beyond a specific time frame.
2023-09-14 17:10:04 +08:00
4fbb25bc55 [Enhancement](function) Support date_trunc(date) and use it in auto partition (#24341)
Support date_trunc(date) and use it in auto partition
2023-09-14 16:53:09 +08:00
Pxl
f303a99250 [Chore](checks) set compile_commands.json path on sonar cloud config (#24374)
set compile_commands.json path on sonar cloud config
2023-09-14 16:52:30 +08:00
b6d7116dea [fix](datetime) fix compare of DatetimeLiteral (#24343)
fix compare of DatetimeLiteral
2023-09-14 16:51:50 +08:00
8e7f7c9566 [fix](profile) move probe time to pull and add LoopGenerateJoin time #24302 2023-09-14 16:41:01 +08:00
7ff76c5a1e [test](Nereids) add normalize sort test (#24230) 2023-09-14 16:33:36 +08:00
Pxl
7b634d8817 [Chore](checks) fix SonarCloud Scan config define (#24370)
fix SonarCloud Scan config define
2023-09-14 16:17:52 +08:00
ace3e79498 [test](Nereids) add test for uncorrelatedApplyFilter #24220
add test for uncorrelatedApplyFilter rule
2023-09-14 16:10:08 +08:00
4efc68a33d [fix](test)disable join reorder for test_bitmap_filter regression test (#23150)
the nereids planner may reorder the join without any statistics info. This could lead to very bad join order that cause the query timeout. This pr disable join reorder for this sql.
2023-09-14 16:05:09 +08:00
ccba5a729a [fix](planner)cast string to float like type should return NULL literal if it fails (#24222) 2023-09-14 15:59:20 +08:00
Pxl
c1b4e132d6 [Chore](checks) add sonar config on cde checks (#24367)
add sonar config on cde checks
2023-09-14 15:54:46 +08:00
40e1c2af45 [fix](nereids)the common type of decimalv2 and decimalv3 shoud be decimalv3 in BinaryArithmetic operator (#24215)
the common type of decimalv2 and decimalv3 shoud be decimalv3 in BinaryArithmetic operator
2023-09-14 15:53:23 +08:00
51a5895464 [Feature](RoutineLoad) Support max filter ratio for routine load (#24035) 2023-09-14 15:30:40 +08:00