Commit Graph

13721 Commits

Author SHA1 Message Date
b9ddcbf729 [feature](merge-cloud) Rewrite code related to IOContext (#24269) 2023-09-15 19:57:58 +08:00
ab69416922 [Bug](pipelineX) fix streaming agg (#24449)
fix streaming agg
2023-09-15 19:22:54 +08:00
c4c4162da2 [fix](test)rename table name to prevent conflicting with other test case (#24391) 2023-09-15 19:11:41 +08:00
Pxl
5286899a71 [Chore](workflow) remove be-ut-clang workflow (#24463)
remove be-ut-clang workflow
2023-09-15 19:09:54 +08:00
07d4769134 [fix](bitmap) fix coredump of bitmap_from_array caused by null array literal (#24404) 2023-09-15 18:36:33 +08:00
0742af70ea [Fix](planner) fix select from inline table return only the first row (#24365) 2023-09-15 18:14:54 +08:00
4ad5845dcc [fix](planner) add if function signature for jsonb data type (#24436) 2023-09-15 17:50:51 +08:00
fa37a8bba8 [opt](stats) remove corresponding col stats status if the loading at the end of analyze task is failed (#24405) 2023-09-15 17:46:48 +08:00
dc0c39f1d8 [Enhance](external)change hive docker to host network and add hive case (#24401)
1. Change the external hive docker network mode from the bridge mode to the host mode to support the external test of the multi-node doris cluster
2. Added more hive test data in various formats
3. Added a test case with hive
2023-09-15 17:46:24 +08:00
4816ca6679 [fix](multi-catalog)fix mc decimal type parse, fix wrong obj location (#24242)
1. mc decimal type need parse correctly by arrow vector method
2. fix wrong obj location if use oss,obs,cosn

Will add test case in another PR
2023-09-15 17:44:56 +08:00
c270a2d89f [typo] (docs)Row storage mode can only be turned on when creating a table (#24257) 2023-09-15 17:40:46 +08:00
699023069d [regression](lateral view) add test case for explode_bitmap (#24421) 2023-09-15 17:30:26 +08:00
8297da56ad [fix](spark load)use low case for expr (#24402)
Use lowercase uniformly for expr.
2023-09-15 17:30:03 +08:00
7fd72351f9 [fix](agg) windown_funnel compatibility issue with multi backends (#24385) 2023-09-15 17:22:47 +08:00
b4d446055a [fix](agg) need to call 'set_version' in nested functions (#24381)
Co-authored-by: yiguolei <676222867@qq.com>
2023-09-15 17:20:20 +08:00
d24f3efd4a [pipelineX](profile) Phase 1: refactor pipelineX detailed profile (#24322) 2023-09-15 16:14:05 +08:00
6dbe07bd3b [Enhancement](inverted index) use conjunction query to accelerate fulltext equal query (#24373) 2023-09-15 15:34:57 +08:00
3c18ed4e86 [test](fix) remove unused test case test_mtmv_ssb_ddl.groovy (#24434)
* forbid: test_mtmv_ssb_ddl

* remove: test_mtmv_ssb_ddl.groovy
2023-09-15 15:02:31 +08:00
a4b62eec63 (enhancement)[fe] Add isMaster() check for FrontendService (#24412)
* In `FrontendServiceImpl` service, the api which need to write editlog
  need add `isMaster()` check
2023-09-15 15:01:08 +08:00
Pxl
5ba2a19cd2 [Chore](check) change sonarcloud github action to c/cpp action (#24433)
change sonarcloud github action to c/cpp action
2023-09-15 14:55:36 +08:00
eb8ecf49bf [fix](planner) should set preserveRootTypes to true when call substituteList method in ExprSubstitutionMap's compose method (#24392)
if set preserveRootTypes to false when calling substituteList, the root cast expr may be lost during substituting. For example, the top cast expr is cast(decimal_col as double), if it's lost, the data type mismatch between plan node and be crashes.
2023-09-15 14:12:06 +08:00
ddc0eb3508 [doc](flink-connector) add json string write (#24422) 2023-09-15 14:08:02 +08:00
08740b47cd [FIX](decimalv3) fix decimalv3 value with leading zeros (#24416)
now we make error if we deal with leading zeros in decimal value , type_precision >= precision will make value overflow and DCHECK will fail , so if here has leading zero we should only make type_precision > precision to make value right
2023-09-15 13:35:20 +08:00
32844b2a5b [fix](java-udf) Fix need to restart BE after replacing the jar package in java-udf (#24372)
Fix need to restart BE after replacing the jar package in java-udf
2023-09-15 13:30:08 +08:00
Pxl
83d5831ae0 [Chore](checks) fix Can't find 'action.yml', 'action.yaml' or 'Dockerfile' (#24424)
fix Can't find 'action.yml', 'action.yaml' or 'Dockerfile'
2023-09-15 13:26:17 +08:00
29fe87982f [improve](outfile) add file_suffix options for outfile (#24334) 2023-09-15 12:58:41 +08:00
3e0933a507 [Enhancement] submit Baidu test cases (#22883) 2023-09-15 11:56:08 +08:00
f57c75f935 [doc](agg_state) add agg_state in data-model (#24348)
add agg_state in data-model
2023-09-15 11:45:21 +08:00
Pxl
15c8ff115c [Chore](checks) run sonar-cloud-java when fe changed (#24420)
run sonar-cloud-java when fe changed
2023-09-15 11:41:35 +08:00
df5b5ae0cb [improvement](create tablet) backend create tablet round robin among disks (#23218)
Backend choose disk by disk available bytes and tablet num. If both are equal, round robin among them.
2023-09-15 11:39:43 +08:00
2976e9d8e5 [imrovement](iterators) reduce memory for union iterators (#24197) 2023-09-15 11:35:56 +08:00
c5e7f55b63 [performance](executor) optimize time_round function (#23058)
optimize time_round function
2023-09-15 10:49:22 +08:00
Pxl
90e434a5dd [Chore](check) change SonarClout Analysis scheduled on push (#24410)
change SonarClout Analysis scheduled on push
2023-09-15 10:42:00 +08:00
dbd7733e02 [feature](regression) Add p2 level test for schema change (#20243) 2023-09-15 10:39:07 +08:00
00bb32cfc0 [opt](nereids) enable two phase partition topn opt #23870
Enable two phase partition topn optimization, instead of original full sort at the second phase.
E.g, partial plan of tpcds q67 is as following and a full sort after exchange will have performance impact, especially if the window column's ndv is very high and the number of window is huge.

------PhysicalTopN
--------filter((rk <= 100))
----------PhysicalWindow
------------PhysicalQuickSort
--------------PhysicalDistribute
----------------PhysicalPartitionTopN
------------------PhysicalProject

Under this scenario, the second phase full sort can be transformed to a global PhysicalPartitionTopN and reduce the cost from full sort. The plan will be optimized to the following:

------PhysicalTopN
--------filter((rk <= 100))
----------PhysicalWindow
------------PhysicalPartitionTopN
--------------PhysicalDistribute
----------------PhysicalPartitionTopN
------------------PhysicalProject
2023-09-15 10:30:34 +08:00
23f01ddf3a [feature](profile) support simply profile (#23377)
A Simplified Version of the Profile

Divided into three levels:
Level 2: The original profile.
Level 1: Instances with identical structures are merged, utilizing concatenation for info strings, and recording the extremum for time types.


Note that currently, this is purely experimental, simplifying the profile on the frontend (you can view profiles at any level).

Subsequently, we will transition the simplification process to the backend. At that point, due to the simplification being done on the backend, viewing profiles at other levels won't be possible.

Due to the issue with the pipeline structure, the active time does not accurately reflect the time of the operators.

```
set enable_simply_profile = false;
set enable_simply_profile = true;
```
2023-09-15 10:25:14 +08:00
320f1e9bbf [improve](routineload) improve show routine load output (#24264) 2023-09-15 10:22:47 +08:00
e0834b2f46 [chore](explain) add annotation in explain string whether nereids is ON #24394 2023-09-15 10:17:17 +08:00
14163df429 [github](config) forbid shell check in pytest directory #24378 2023-09-15 10:15:08 +08:00
9c681692bd Revert "[fix] fix http_stream retry mechanism (#23969)" (#24407)
This reverts commit 05e365ea137eb8c92b8e7eedc7d1435e83f065ae.
2023-09-15 10:07:53 +08:00
c5ef6cfea2 [fix](Table-Valued Function) fix be core when user sepcified empty column_separator using hdfs tvf (#24369) 2023-09-14 23:19:48 +08:00
5ba1f62da8 [enhancement](Nereids) make stats unchanged (#23737)
make stats unchanged when explore plan
2023-09-14 22:18:54 +08:00
66bd2a4862 [test](Nereids) add test push down filter (#24250)
Add test for pushDownFilterThroughProject
2023-09-14 22:13:41 +08:00
d4756d3118 [feature](Nereids): fold Cast(s as date/datetime) on FE (#24353)
cast("20210101" as Date) -> DateLiteral(2021, 1, 1)
2023-09-14 22:08:26 +08:00
f61e6483bf [enhancement](broker-load) support compress type for old broker load, and split compress type from file format (#23882) 2023-09-14 21:42:28 +08:00
0488c87a38 [typo][doc]modify broken link. (#23988) 2023-09-14 21:41:40 +08:00
05e365ea13 [fix] fix http_stream retry mechanism (#23969)
Co-authored-by: yiguolei <676222867@qq.com>
2023-09-14 21:41:11 +08:00
07720d3ff9 [feature](replica version) Add admin set replica version statement (#23706) 2023-09-14 21:12:00 +08:00
d20365cdcf [fix](transaction) fix publish txn fake succ (#24273) 2023-09-14 21:04:59 +08:00
0c30720c99 [Hotfix](merge-on-write) Fix when DeltaWriter don't report num_filtered_rows in TTabletCommitInfo (#24352) 2023-09-14 20:58:08 +08:00