34b048a2bd
[fix](nereids) update outer join estimation #21126
...
the row count of left outer join should be no less than left child row count.
2023-06-25 21:37:55 +08:00
af2b67e65a
[Fix](multi-catalog) Invalidate cache when enable auto refresh catalog. ( #21070 )
...
The default value of RefreshCatalogStmt.invalidCache is false now, but the RefreshManager.RefreshTask does not invoke RefreshCatalogStmt.analyze() so it will not invalidate the cache. This pr mainly fix this problem
2023-06-25 19:14:44 +08:00
638aa41988
[fix](planner) fix push filter through agg #21080
...
In the previous implementation, the check for groupby exprs was ignored. Add this necessary check to make sure it would work
You could reproduce it by runnning belowing sql:
CREATE TABLE t_push_filter_through_agg (col1 varchar(11451) not null, col2 int not null, col3 int not null)
UNIQUE KEY(col1)
DISTRIBUTED BY HASH(col1)
BUCKETS 3
PROPERTIES(
"replication_num"="1"
);
CREATE VIEW `view_i` AS
SELECT
`b`.`col1` AS `col1`,
`b`.`col2` AS `col2`
FROM
(
SELECT
`col1` AS `col1`,
sum(`cost`) AS `col2`
FROM
(
SELECT
`col1` AS `col1`,
sum(CAST(`col3` AS INT)) AS `cost`
FROM
`t_push_filter_through_agg`
GROUP BY
`col1`
) a
GROUP BY
`col1`
) b;
SELECT SUM(`total_cost`) FROM view_a WHERE `dt` BETWEEN '2023-06-12' AND '2023-06-18' LIMIT 1;
2023-06-25 19:14:20 +08:00
69d5adaee3
[Improvement](doc) improve ngram and inverted index documents #21091
2023-06-25 19:13:41 +08:00
ee2492dd78
[typo](doc)fix delete table associate to other table only support unique model ( #21129 )
...
Co-authored-by: smallhibiscus <844981280>
2023-06-25 19:04:27 +08:00
55e7af1e31
[fix](test) fix two case bug #21124
2023-06-25 18:53:20 +08:00
b6c9feb458
[fix](nereids) check table privilege when it's needed ( #21130 )
...
check privilege on LogicalOlapScan, LogicalEsScan, LogicalFileScan and LogicalSchemaScan
2023-06-25 18:35:39 +08:00
46f0295b78
[feature](load-refactor-with-tvf) S3 load with S3 tvf and native insert ( #19937 )
2023-06-25 17:45:31 +08:00
771b0cbb4c
[fix](stats) Update analyze task execute time ( #21026 )
...
Before this PR last_execute_time of pending analyze jobs would be 1970-01-01, you can reproduce it by run show analyze
2023-06-25 15:52:33 +08:00
cf66280e60
[opt](stats) Sampling when aggregate column stats ( #21020 )
...
In the previous implementation, when aggregating partition statistics into column statistics, the calculation of distinct values (ndv) for the entire column was performed without using sampling, resulting in reduced efficiency of the sampling process.
Before this PR analyze below table which has 1000000 lines would cost 5.75sec, after this PR, it would cost 3.39sec.
```sql
CREATE TABLE IF NOT EXISTS `duplicate_all` (
`k3` int(11) null comment "",
`k0` boolean null comment "",
`k1` tinyint(4) null comment "",
`k2` smallint(6) null comment "",
`k4` bigint(20) null comment "",
`k5` decimalv3(9, 3) null comment "",
`k6` char(36) null comment "",
`k10` date null comment "",
`k11` datetime null comment "",
`k7` varchar(64) null comment "",
`k8` double null comment "",
`k9` float null comment "",
`k12` string null comment "",
`k13` largeint(40) null comment ""
) engine=olap
DUPLICATE KEY(`k3`)
DISTRIBUTED BY HASH(`k3`) BUCKETS 5 properties("replication_num" = "3")
```
2023-06-25 15:52:01 +08:00
dd99468b8f
[fix](stats) Fix jdbc timeout with multiple FE when execute analyze table ( #21115 )
...
SQL may forward to master to execute when connecting to follower node, the result should be set to `StmtExecutor#proxyResultSet`
Before this PR, in above scenario , submit analyze sql by mysql client/jdbc whould return get malformed packet/ Communication failed.
2023-06-25 15:49:36 +08:00
76bdcf1d26
[improvement](pipeline) task group scan entity ( #19924 )
2023-06-25 14:43:35 +08:00
80d54368e0
[minor](Nereids) replace some nullable field to Optional ( #20967 )
2023-06-25 12:02:25 +08:00
6896776034
[test](regression) update some case in p2 ( #21094 )
...
update some case in p2
2023-06-25 11:16:56 +08:00
207bc53b06
[functionpushdown](performance) move function pushdown as default false since its performance is not good ( #21111 )
...
set enable function pushdown default to false.
enable it in fuzzy mode to test this feature.
We should remove function pushdown in the future since we already have common expr pushdown.
Co-authored-by: yiguolei <yiguolei@gmail.com >
2023-06-25 10:36:20 +08:00
20b92b0812
[Feature](log)friendly hint for creating table failed ( #20617 )
2023-06-25 10:02:26 +08:00
d49c412c59
[Feature](multi-catalog) Add hdfs benchmark tools. ( #21074 )
2023-06-25 09:35:27 +08:00
601120db04
[Bug](pipeline) access map may cause coredump in sink buffer ( #21108 )
2023-06-24 23:03:59 +08:00
5aa16e84bf
[fix](catalog) do not call makeSureInitialized when create table from hms meta event ( #21104 )
...
In this PR, I remove the `makeSureInitialized()` call in `createTable()` method, because it is wrong and useless.
And also rename the methed's name to make it more clear.
2023-06-24 21:50:36 +08:00
691a988c97
[enhancement](merge-on-write) add async publish task when version is discontinuous for merge on write table when clone ( #21025 )
...
version discontinuity may occur when clone. To deal with this case, add async publish task when version is discontinuous.
2023-06-22 21:50:14 +08:00
fa3bb2eabe
[Bug](materialized-veiw) fix error happens when parsing create materialized view stmt #21095
2023-06-22 15:58:32 +08:00
eb6202e8be
[minor](fe) remove several unnecessary codes ( #21046 )
...
1. The class 'ExternalDatabase' has implemented the 'GsonPostProcessable' interface, so
there is redundant codes in some subclass of 'ExternalDatabase'.
2. A LOG object is not used in this file.
2023-06-22 15:29:25 +08:00
6a2226867c
[fix](test) load case bug in tpch_sf10_unique_p2 #21093
2023-06-22 15:19:46 +08:00
37c9a08e56
[Bug] The PID_DIR variable in the Doris stop script does not follow the conf file ( #20881 )
2023-06-22 10:26:26 +08:00
d5455dd04b
[fix](regression-test) test case variable scope bug ( #21071 )
2023-06-22 09:07:44 +08:00
bc6692a1b8
[typo](doc) add automatic service start ( #20974 )
...
* add automatic service start
* add automatic service start
* add docs sidebars
* Update automatic-service-start.md
2023-06-22 08:45:09 +08:00
9f0aa8a9de
[fix](fuzzy)nereids and pipeline config changed by fuzzy in non-pipeline env. ( #21092 )
...
* fix: nereids and pipeline config changed by fuzzy in non-pipeline env.
* fix: format
* fix: format
2023-06-22 08:36:19 +08:00
a33521b2ce
[enhancement](exchange) add filter for exchange node in BE ( #21087 )
2023-06-22 01:04:47 +08:00
b192082b62
[Improve](load)Solve the problem of RoutineLoadTaskScheduler idling when there is no data ( #20986 )
...
Since the polling interval is 0, the CPU will be polled all the time when there is no data
Before and after comparison test, the CPU usage time is reduced by 2000 times
2023-06-22 00:41:45 +08:00
fff308352f
[fix](nereids)the microseconds value is wrong when create datatimev2 literal from LocalDateTime ( #21089 )
...
* [fix](nereids)the microseconds value is wrong when create datatimev2 literal from LocalDateTime
* fix code style
2023-06-22 00:40:53 +08:00
49bbe88327
[fix](log) fix the too large warning log of BE ( #21027 )
2023-06-22 00:39:04 +08:00
3dfeee3946
[fix](typesystem) fix wrong return type argument cause type check fail ( #21082 )
2023-06-22 00:04:46 +08:00
2c9bdd64fa
[fix](memory) arena support memory reuse after clear() ( #21033 )
2023-06-21 23:27:21 +08:00
e060ffab96
[Fix](cooldown) Fix incorrect judgement of isDropTableOrPartition ( #21084 )
2023-06-21 23:00:58 +08:00
2ce8cfbebd
[profile](sort) add some metrics in profile ( #21056 )
2023-06-21 22:57:46 +08:00
661e1ae7c5
[fix](memory) no switch bthread context in UBSAN compile ( #21064 )
...
When UBSAN is compiled, all memory will be tracked to the orphan (unknown) mem tracker, and the bthread context and mem tracker will no longer be switched.
The supplementary fixes are as follows: #20999
2023-06-21 21:14:07 +08:00
b2c4e51be1
[fix](load) delete lazy open DCheck when unkown load id ( #21083 )
2023-06-21 20:42:31 +08:00
18a0824eb3
[fix](compaction)Modify time series compaction policy default config ( #21079 )
2023-06-21 20:29:58 +08:00
8b561cfb03
[fix](nereids)create datev2 and datetimev2 literal if enable_date_conversion is true ( #21065 )
2023-06-21 20:29:36 +08:00
442a734ef5
[improvement](config) update be config max_runnings_transactions_per_txn_map default value ( #21060 )
2023-06-21 20:29:13 +08:00
6ac0bfeceb
[Feature](inverted index) add unicode parser for inverted index ( #21035 )
2023-06-21 20:14:06 +08:00
cc53391c9a
Revert "[feature](merge-on-write) enable merge on write by default (#… ( #21041 )
2023-06-21 18:36:46 +08:00
2beed11256
[Bug](streamload) fix inconsistent load result of be and fe ( #20950 )
2023-06-21 18:12:51 +08:00
84b97860a1
[fix](memory) Fix memory exceed limit and query has been canceled, Allocator will block 100ms ( #20959 )
2023-06-21 17:35:19 +08:00
8bcd42d3f6
[test](regression) update some case in brown_p2 #21037
2023-06-21 16:25:07 +08:00
85ce6a22c0
[enhancement](merge-on-write) some misc optimizations ( #21039 )
2023-06-21 16:16:06 +08:00
4766e75d4e
[test](regression) add analyze after stream load ( #21066 )
2023-06-21 16:03:46 +08:00
4d84cd8ca1
Revert "Revert "[Test](regression) CCR syncer thrift interface regression test ( #20935 )" ( #20990 )" ( #21022 )
...
This reverts commit 2a294801f1324a999570158eea3224239eefbb29.
2023-06-21 15:20:21 +08:00
b65b821813
[enhancement](pk) add bvar stating cached io ( #20977 )
2023-06-21 15:02:10 +08:00
c5560b8f93
[fix](load) segcompaction does not signal waiters when an error hanppens ( #21043 )
...
This leads to a deadlock.
2023-06-21 14:56:34 +08:00
