Commit Graph

2976 Commits

Author SHA1 Message Date
ee8dffbfb7 [meta](recover) change dropInfo and RecoverInfo to GSON (#13830) 2022-11-02 13:32:46 +08:00
d5becdb4a1 [fix](dynamic-partition) fix wrong check of replication num (#13755) 2022-11-02 12:55:33 +08:00
wxy
947e67fa76 [enhancement](test) retry start be or fe when port has been bind. (#13860)
Co-authored-by: wangxiangyu@360shuke.com <wangxiangyu@360shuke.com>
2022-11-02 08:42:35 +08:00
0eeb4d2881 [minor](log) remove some e.printStackTrace() (#13870) 2022-11-02 08:42:10 +08:00
7fedfdcf6a [fix](spark load)The where condition does not take effect when spark load loads the file (#13803) 2022-11-01 23:01:45 +08:00
287a739510 [javaudf](string) Fix string format in java udf (#13854) 2022-11-01 21:25:12 +08:00
7f34698eef [enhancement](Nereids) use join estimation v2 only when stats derive v2 is enable (#13845)
join estimation V2 should be invoked when enableNereidsStatsDeriveV2=true
2022-11-01 20:38:39 +08:00
f0c9867af3 [fix](nereids) map literal to double in FilterSelectivityCalculator (#13776)
fix literal to double bug: all literal type implements getDouble() function
2022-11-01 20:20:44 +08:00
01f9f8ad43 [enhancement](Nereids) add merge project rule to column prune rule set (#13835)
when we do column prune, we add project on child plan. If child plan is Project. we need to merge them.
2022-11-01 20:17:53 +08:00
61c817f4cc [feature](syntax) support SELECT * EXCEPT (#13844)
* [feature](syntax) support SELECT * EXCEPT: add regression test
2022-11-01 19:41:25 +08:00
1eef986e75 [feature](nereids) add rule for semi/anti join exploration, when there is project between them (#13756) 2022-11-01 19:07:25 +08:00
c14277e587 [fix](analytic) fix coredump cause by empty analytic parameter types (#13808)
* fix fe compile error
2022-11-01 17:25:36 +08:00
83e55cade8 [feature](Nereids): add rule for matching plan into HyperGraph. (#13805) 2022-11-01 14:57:25 +08:00
34e68a41dd [enhancement](explain) add cardinality to explain string and explain graph (#13720)
1. set cardinality when translate Nereids plan to legacy planner's plan
2. print cardinality when use EXPLAIN GRAPH
2022-11-01 11:43:21 +08:00
b27714542d [fix](planner) infer predicate could generate predicates in another scope (#13691)
* [fix](planner) infer predicate could generate predicates in another scope
2022-11-01 09:03:41 +08:00
36a47dfe16 [enhancement](Nereids): use ImmutableList explicitly in Plan (#13817) 2022-10-31 20:23:30 +08:00
18be77af64 [fix](nereids) query cannot execution when both nereids enable and fallback to legacy planner are set to false (#13787)
when enable_nereids_planner=false and enable_fallback_to_origin=false, FE throws exception for all select statement.
Expected: when enable_nereids_planner=false, all valid query execution success
2022-10-31 19:02:01 +08:00
ba177a15cb [feature-wip](recover) new recover ddl and support show catalog recycle bin (#13067) 2022-10-31 17:44:56 +08:00
ceb7b60a64 [fix](Nereids) update immutable LogicalAggregate attribute by mistake (#13740) 2022-10-31 14:11:55 +08:00
53e5f3939e [fix](plan)result exprs should be substituted in the same way as agg exprs (#13744)
* [fix](cast)ignore implicit cast when comparing two exprs

* fix fe ut
2022-10-31 10:19:32 +08:00
61b7c2c96c [fix](join) fix incorrect result when using anti join with other join predicates (#13743) 2022-10-31 09:51:34 +08:00
efe813ba60 [fix](test)(explain) add full qualified name for scan node explain string (#13777)
1.
In the "explain" result of SQL, the table name in `ScanNode` should be full qualified with dbname.
And for olap scan node, the selected index name should not be "null".

2.
Remove `tpch_sf1_p1/tpch_sf1/nereids/` in regression test, it will be fixed later.
2022-10-30 13:24:48 +08:00
2a5d3dbb6e feat(nereids): draw hyper graph by graphviz (#13749) 2022-10-28 17:23:35 +08:00
e0667b297f [feature-wip](multi-catalog) reuse hdfsFs and decode parquet values in batch (#13688)
PR(https://github.com/apache/doris/pull/13404) introduced that ParquetReader
will break up batch insertion when encountering null values, which leads to the bad performance
compared to OrcReader.
So this PR has pushed null map into decode function, reduce the time of virtual function call
when encountering null values.

Further more, reuse hdfsFS among file readers to reduce the time of building connection to hdfs.
2022-10-28 15:52:52 +08:00
Pxl
2fab0c45c7 [Feature](runtime-filter) add runtime filter breaking change adapt (#13246)
add runtime filter breaking change adapt
2022-10-28 10:59:28 +08:00
45b31506c7 [improvement](delete) support delete from partitioned table without partition specified (#13533)
Support delete from partitioned table without partition specified in [DELETE] stmt.

## Usage
If it is a partitioned table, you can specify a partition.
If not specified, Doris will infer partition from the given conditions.
In two cases, Doris cannot infer the partition from conditions:
1) the conditions do not contain partition columns;
2) The operator of the partition column is `not in`.
When a partition table does not specify the partition,
or the partition cannot be inferred from the conditions,
the session variable `delete_without_partition` needs to be `true`
to make delete statement be applied to all partitions.

## Test case
Test case is added in `regression-test/suites/delete_p0/test_delete_from_partition.groovy`,
user can delete from partitioned table without partition specified now.
2022-10-27 21:32:45 +08:00
ec86e9c9b2 [feature-wip][MTMV] The schedule framework for the MTMV (#13147)
Design document: https://github.com/apache/doris/issues/13146
2022-10-27 11:37:24 +08:00
0e70d681d9 [feature](Nereids): Construct join graph (#13679)
* feat: add hypergraph and its api

* feat: add visulization api

Signed-off-by: xiejiann <jianxie0@gmail.com>

* remove unused code

Signed-off-by: xiejiann <jianxie0@gmail.com>

* fix format

Signed-off-by: xiejiann <jianxie0@gmail.com>

* remove unused test

Signed-off-by: xiejiann <jianxie0@gmail.com>

* remove unused tests

Signed-off-by: xiejiann <jianxie0@gmail.com>

* format

Signed-off-by: xiejiann <jianxie0@gmail.com>

Signed-off-by: xiejiann <jianxie0@gmail.com>
2022-10-27 11:32:31 +08:00
2697f72d77 [Improvement][SET-PROPERTY] Support for set query_timeout property (#13444) 2022-10-27 10:03:39 +08:00
7557980d64 [improvement](regression-test) avoid query empty result after loading finished (#13682)
When running regression test, we always found that the query return empty result after loading finished,
even if we call "sync" before the query.
This is because for `stream load`, the load task result will be returned immediately after the txn's status changed to VISIBLE,
but before writing the edit log.
So if we do the query right after we got the load task result, it is possible that we can not see the latest loaded data.

Same issue with `insert` operation
2022-10-27 09:47:18 +08:00
5bd66243ee [minor](log) remove some unused logs (#13689)
1. When running regression test with specific suites or group, do not print other suite name or file name
2. Remove unused alter table job log.
2022-10-27 09:37:32 +08:00
3c95106d45 [Bug](jdbc) Fix memory leak for JDBC datasource (#13657) 2022-10-27 00:02:25 +08:00
ddb27b9c3f nereids use decimal(27,9) (#13678) 2022-10-26 21:37:24 +08:00
f4c8d4ce85 [feature](nereids) estimate plan cost by column ndv and table row count (#13375)
In this version, we use column ndv information to estimate plan cost.

This is the first version, covers TPCH queries.
2022-10-26 20:35:10 +08:00
bed759b3f5 [Fix](array-type) support CTAS for ARRAY column from collect_list and collect_set (#13627)
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2022-10-26 19:42:15 +08:00
0841c5bf28 [Bugfix](manager) fix query profile key incompatible with old versions (#13596) 2022-10-26 14:27:58 +08:00
3548d0b824 [fix](statistics) fix cross join statistics exception (#13645) 2022-10-26 14:10:57 +08:00
c418bbd2d1 [feature-wip](new-scan) support Json reader (#13546)
Issue Number: close #12574
This pr adds `NewJsonReader` which implements GenericReader interface to support read json format file.

TODO:
1. modify `_scann_eof` later.
2. Rename `NewJsonReader` to `JsonReader` when `JsonReader` is deleted.
2022-10-26 12:52:21 +08:00
44c9163b3c [Fix](multi-catalog)Fix partition external table query bug. (#13535)
The index for external table columns from path is incorrect in new scanner. This is a fix for it.
e.g. In the next query, nation and city columns are from path
```
mysql> select nation, city, count(*) from parquet_two_part group by nation, city;
+--------+------------+----------+
| nation | city       | count(*) |
+--------+------------+----------+
| cn     | beijing    |  1199969 |
| cn     | shanghai   |  1199771 |
| jp     | tokyo      |   599715 |
| rus    | moscow     |   600659 |
| us     | chicago    |  1199805 |
| us     | washington |  1201296 |
+--------+------------+----------+
6 rows in set (0.39 sec)
```
2022-10-26 12:47:37 +08:00
15130c469f [fix](planner) cannot recogonize column's table when analyze rewrite expr (#13597)
We save mv column with alias as table name, and search it with original table name.
2022-10-26 11:15:48 +08:00
e5b33abd3c [fix](planner) inlineView alias error (#13600) 2022-10-26 10:14:04 +08:00
e385cb063c [improvement](config) allow to modify the master-only configuration of non-master nodes (#13558)
Non-master nodes may switch to master, so we should allow the master-only configuration of all fe nodes to be modified.
2022-10-26 10:00:12 +08:00
c709998faa [improvement][refactor](mysql) remove old mysql server and add keep alive option (#13663)
* [improvement][refactor](mysql) remove old mysql server and add keep alive option
2022-10-26 09:38:33 +08:00
a02a56eb38 [fix](postgresql) fix postgresql cann't find table (#13550) 2022-10-26 09:30:28 +08:00
e00734348b [Chore](regression) Fix wrong result for decimal (#13644) 2022-10-26 09:24:46 +08:00
c486d9746d [fix](broker) fix bug when broker load with s3a (#13650)
Signed-off-by: nextdreamblue <zxw520blue1@163.com>

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2022-10-26 09:22:38 +08:00
9691db7918 [Enhancement](metrics) add more metrics (#11693)
* Add `AutoMappedMetric` to measure dynamic object.
* Add query instance and rpc metrics
* Add thrift rpc metrics
* Add txn metrics
* Reorganize metrics init routine.

Co-authored-by: 迟成 <chicheng@meituan.com>
2022-10-26 08:31:03 +08:00
17ba40f947 [feature-wip](CN Node)Support compute node (#13231)
Introduce the node role to doris, and the table creation and tablet scheduler will control the storage only assign to the BE nodes.
2022-10-25 21:44:33 +08:00
cb39671a73 [fix](policy) Add readlock for show policy (#13497)
Add readlock for show policy resolve ConcurrentModificationException
2022-10-25 21:42:40 +08:00
d6c3470c8d [feature](Nereids) support materialized index selection (#13416)
This PR unified the selection of rollup index and materialized view index into uniform logic, which is called selecting materialized index. 

Main steps:

### Find candidate indexes
1. When aggregate is present, it's handled in `SelectMaterializedIndexWithAggregate`.  The base index and indexes that could use pre-aggregation should be used. The pre-aggregation status is determined by aggregation function, grouping expression, and pushdown predicates.
2. When aggregate is not on top of scan node, it's handled in `SelectMaterializedIndexWithoutAggregate`. The base index and indexes that have all the key columns could be used.

### Filter and order the candidate indexes
1. filter indexes that contain all the required output scan columns.
2. filter indexes that could match prefix index most.
3. order the result index by row count, column count, and index id.
2022-10-25 19:25:58 +08:00