Commit Graph

6888 Commits

Author SHA1 Message Date
bad950136d [chore](build) Pass the compile flag -Wno-unused-but-set-variable on demand (#13716)
There are some issues with the compile flag `-Wno-unused-but-set-variable` for clang.
1. `-Wno-unused-but-set-variable` should be set when building source by clang-15 on Linux. (#13000 #13016)
2. On macOS Monterey, Apple Clang 13 may treat it as a unknown warning option and the compilation process may interrupt.

This PR introduces a better way to make this compile flag more portable.
1. Test whether the compiler recognizes this flag.
2. Add this flag if the compiler recognizes it.
2022-10-27 15:18:28 +08:00
738da0b139 [bugfix](join) inner join return wrong result (#13608)
* bug fix for vhash join

* add regression test

Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2022-10-27 11:48:41 +08:00
ec86e9c9b2 [feature-wip][MTMV] The schedule framework for the MTMV (#13147)
Design document: https://github.com/apache/doris/issues/13146
2022-10-27 11:37:24 +08:00
0e70d681d9 [feature](Nereids): Construct join graph (#13679)
* feat: add hypergraph and its api

* feat: add visulization api

Signed-off-by: xiejiann <jianxie0@gmail.com>

* remove unused code

Signed-off-by: xiejiann <jianxie0@gmail.com>

* fix format

Signed-off-by: xiejiann <jianxie0@gmail.com>

* remove unused test

Signed-off-by: xiejiann <jianxie0@gmail.com>

* remove unused tests

Signed-off-by: xiejiann <jianxie0@gmail.com>

* format

Signed-off-by: xiejiann <jianxie0@gmail.com>

Signed-off-by: xiejiann <jianxie0@gmail.com>
2022-10-27 11:32:31 +08:00
d388de6c11 [Enhancement](threadpool) print thread pool name on error (#13706) 2022-10-27 10:49:18 +08:00
c874931ac8 [fix](join)output all value from no-null side of outer join (#13655)
* [fix](joinoutput all value from no-null side of outer join

* add regression test
2022-10-27 10:48:36 +08:00
2697f72d77 [Improvement][SET-PROPERTY] Support for set query_timeout property (#13444) 2022-10-27 10:03:39 +08:00
7557980d64 [improvement](regression-test) avoid query empty result after loading finished (#13682)
When running regression test, we always found that the query return empty result after loading finished,
even if we call "sync" before the query.
This is because for `stream load`, the load task result will be returned immediately after the txn's status changed to VISIBLE,
but before writing the edit log.
So if we do the query right after we got the load task result, it is possible that we can not see the latest loaded data.

Same issue with `insert` operation
2022-10-27 09:47:18 +08:00
ffcb2f8525 [opt](exec) Replace get_utf8_byte_length function by array (#13664) 2022-10-27 09:46:41 +08:00
5bd66243ee [minor](log) remove some unused logs (#13689)
1. When running regression test with specific suites or group, do not print other suite name or file name
2. Remove unused alter table job log.
2022-10-27 09:37:32 +08:00
3e8cd0c669 [typo](doc) Add the description of json HDFS broker load (#13683)
Add the instruction of HDFS broker load with json format file.
2022-10-27 09:36:57 +08:00
d2262bc8fb [docs]fix 404 (#13695)
[docs]fix 404
2022-10-27 08:49:36 +08:00
3c95106d45 [Bug](jdbc) Fix memory leak for JDBC datasource (#13657) 2022-10-27 00:02:25 +08:00
0134e9d2f4 [Improvement](runtime filter) Reduce merging time for bloom filter (#13668) 2022-10-27 00:02:05 +08:00
06e433e14a [fix](cmake)fix cmake error (#13637)
fix cmake error if variables(${LIB_JVM}) is ""
2022-10-26 21:38:50 +08:00
ddb27b9c3f nereids use decimal(27,9) (#13678) 2022-10-26 21:37:24 +08:00
f4c8d4ce85 [feature](nereids) estimate plan cost by column ndv and table row count (#13375)
In this version, we use column ndv information to estimate plan cost.

This is the first version, covers TPCH queries.
2022-10-26 20:35:10 +08:00
bed759b3f5 [Fix](array-type) support CTAS for ARRAY column from collect_list and collect_set (#13627)
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2022-10-26 19:42:15 +08:00
c5559877b4 [typo](docs)fix docs 404 link (#13677) 2022-10-26 14:56:47 +08:00
0841c5bf28 [Bugfix](manager) fix query profile key incompatible with old versions (#13596) 2022-10-26 14:27:58 +08:00
65aa863dcf [Bugfix](bitmap) Fix to_bitmap_with_check function symbol is incorrect (#13667)
* [Bugfix](bitmap) Fix to_bitmap_with_check function symbol is incorrect
2022-10-26 14:27:38 +08:00
3548d0b824 [fix](statistics) fix cross join statistics exception (#13645) 2022-10-26 14:10:57 +08:00
c418bbd2d1 [feature-wip](new-scan) support Json reader (#13546)
Issue Number: close #12574
This pr adds `NewJsonReader` which implements GenericReader interface to support read json format file.

TODO:
1. modify `_scann_eof` later.
2. Rename `NewJsonReader` to `JsonReader` when `JsonReader` is deleted.
2022-10-26 12:52:21 +08:00
44c9163b3c [Fix](multi-catalog)Fix partition external table query bug. (#13535)
The index for external table columns from path is incorrect in new scanner. This is a fix for it.
e.g. In the next query, nation and city columns are from path
```
mysql> select nation, city, count(*) from parquet_two_part group by nation, city;
+--------+------------+----------+
| nation | city       | count(*) |
+--------+------------+----------+
| cn     | beijing    |  1199969 |
| cn     | shanghai   |  1199771 |
| jp     | tokyo      |   599715 |
| rus    | moscow     |   600659 |
| us     | chicago    |  1199805 |
| us     | washington |  1201296 |
+--------+------------+----------+
6 rows in set (0.39 sec)
```
2022-10-26 12:47:37 +08:00
15130c469f [fix](planner) cannot recogonize column's table when analyze rewrite expr (#13597)
We save mv column with alias as table name, and search it with original table name.
2022-10-26 11:15:48 +08:00
e5b33abd3c [fix](planner) inlineView alias error (#13600) 2022-10-26 10:14:04 +08:00
e385cb063c [improvement](config) allow to modify the master-only configuration of non-master nodes (#13558)
Non-master nodes may switch to master, so we should allow the master-only configuration of all fe nodes to be modified.
2022-10-26 10:00:12 +08:00
c709998faa [improvement][refactor](mysql) remove old mysql server and add keep alive option (#13663)
* [improvement][refactor](mysql) remove old mysql server and add keep alive option
2022-10-26 09:38:33 +08:00
295d887cf5 [improvement](thread) set name for priority thread pool (#13552) 2022-10-26 09:32:15 +08:00
a02a56eb38 [fix](postgresql) fix postgresql cann't find table (#13550) 2022-10-26 09:30:28 +08:00
bd2b803019 [opt](tpch) opt the q9 of tpch to change instance to 8 (#13656) 2022-10-26 09:28:51 +08:00
e00734348b [Chore](regression) Fix wrong result for decimal (#13644) 2022-10-26 09:24:46 +08:00
b13061360f [enhancement](chore) build fe-common when build java-udf (#13647)
* [enhancement](chore) Enhance build script compatibility

* remove duplicate fix
2022-10-26 09:23:21 +08:00
c486d9746d [fix](broker) fix bug when broker load with s3a (#13650)
Signed-off-by: nextdreamblue <zxw520blue1@163.com>

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2022-10-26 09:22:38 +08:00
2563dcca95 [fix](load) fix core dump when get_memtable_consumption_inflush (#13629)
If delta writer is not inited, _flush_token might be nullptr.
2022-10-26 09:20:33 +08:00
9691db7918 [Enhancement](metrics) add more metrics (#11693)
* Add `AutoMappedMetric` to measure dynamic object.
* Add query instance and rpc metrics
* Add thrift rpc metrics
* Add txn metrics
* Reorganize metrics init routine.

Co-authored-by: 迟成 <chicheng@meituan.com>
2022-10-26 08:31:03 +08:00
458cb8f072 [chore](macOS) Support macOS Ventura (13.0) (#13626)
We need the tool `makeinfo` to build third parties on macOS Ventura (13.0).
2022-10-26 08:15:21 +08:00
6f18726f01 [improvement](test) add sync for test_agg_keys_schema_change_datev2 (#13643)
1. add "sync" to avoid some potential meta sync problem when running regression test on multi-node cluster
2. Use /tmp dir as dest dir of outfile test, to avoid "No such file or directory" error.
2022-10-25 22:29:05 +08:00
17ba40f947 [feature-wip](CN Node)Support compute node (#13231)
Introduce the node role to doris, and the table creation and tablet scheduler will control the storage only assign to the BE nodes.
2022-10-25 21:44:33 +08:00
cb39671a73 [fix](policy) Add readlock for show policy (#13497)
Add readlock for show policy resolve ConcurrentModificationException
2022-10-25 21:42:40 +08:00
4db18c139c [fix](thirdparty) g++-11: error: unrecognized command-line option '-m… (#13607)
When compiling third-party hdfs on aarch64 machine,
got error: g++-11: error: unrecognized command-line option '-msse4.2'
2022-10-25 21:35:31 +08:00
d6c3470c8d [feature](Nereids) support materialized index selection (#13416)
This PR unified the selection of rollup index and materialized view index into uniform logic, which is called selecting materialized index. 

Main steps:

### Find candidate indexes
1. When aggregate is present, it's handled in `SelectMaterializedIndexWithAggregate`.  The base index and indexes that could use pre-aggregation should be used. The pre-aggregation status is determined by aggregation function, grouping expression, and pushdown predicates.
2. When aggregate is not on top of scan node, it's handled in `SelectMaterializedIndexWithoutAggregate`. The base index and indexes that have all the key columns could be used.

### Filter and order the candidate indexes
1. filter indexes that contain all the required output scan columns.
2. filter indexes that could match prefix index most.
3. order the result index by row count, column count, and index id.
2022-10-25 19:25:58 +08:00
Pxl
bd884d3298 [Chore](build) add a environment variable DISABLE_JAVA_UDF (#13588) 2022-10-25 17:46:56 +08:00
2c70b17a47 [Del](vec) Support in predicate in delete condition of or and (#13587) 2022-10-25 17:33:35 +08:00
1741a20689 [opt](planer) remove unless cast of avg function (#13593) 2022-10-25 17:33:02 +08:00
f209b7ab6e [fix](Nereids) add exchange node check between local and global agg in plan translator (#12913)
### table schema
CREATE TABLE `t1` (
  `k1` int(11) NULL,
  `v1` int(11) NULL
) ENGINE=OLAP
DUPLICATE KEY(`k1`, `v1`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`k1`)
BUCKETS 3
PROPERTIES('replication_num'='1')

### query
select k1,count(distinct v1+1) from t1 group by k1;

### error
java.lang.ClassCastException: org.apache.doris.planner.OlapScanNode cannot be cast to org.apache.doris.planner.AggregationNode
2022-10-25 16:55:29 +08:00
e103531e69 [fix](sort)order by constant expr bug (#13613)
Issue Number: close (#13350)
2022-10-25 16:43:18 +08:00
b85c78ee00 [fix](regression) add 'if not exists' to 'create table' to support parallel test (#13576) (#13578)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2022-10-25 16:37:07 +08:00
78278f5943 [chore](be version) Check BE version by script (#13594)
Check BE version by script
2022-10-25 16:20:38 +08:00
235c105554 [feature-array](array-type) Add array function array_enumerate (#13612)
Add array function array_enumerate
2022-10-25 15:12:11 +08:00