Commit Graph

8276 Commits

Author SHA1 Message Date
3c95106d45 [Bug](jdbc) Fix memory leak for JDBC datasource (#13657) 2022-10-27 00:02:25 +08:00
0134e9d2f4 [Improvement](runtime filter) Reduce merging time for bloom filter (#13668) 2022-10-27 00:02:05 +08:00
06e433e14a [fix](cmake)fix cmake error (#13637)
fix cmake error if variables(${LIB_JVM}) is ""
2022-10-26 21:38:50 +08:00
ddb27b9c3f nereids use decimal(27,9) (#13678) 2022-10-26 21:37:24 +08:00
f4c8d4ce85 [feature](nereids) estimate plan cost by column ndv and table row count (#13375)
In this version, we use column ndv information to estimate plan cost.

This is the first version, covers TPCH queries.
2022-10-26 20:35:10 +08:00
bed759b3f5 [Fix](array-type) support CTAS for ARRAY column from collect_list and collect_set (#13627)
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2022-10-26 19:42:15 +08:00
c5559877b4 [typo](docs)fix docs 404 link (#13677) 2022-10-26 14:56:47 +08:00
0841c5bf28 [Bugfix](manager) fix query profile key incompatible with old versions (#13596) 2022-10-26 14:27:58 +08:00
65aa863dcf [Bugfix](bitmap) Fix to_bitmap_with_check function symbol is incorrect (#13667)
* [Bugfix](bitmap) Fix to_bitmap_with_check function symbol is incorrect
2022-10-26 14:27:38 +08:00
3548d0b824 [fix](statistics) fix cross join statistics exception (#13645) 2022-10-26 14:10:57 +08:00
c418bbd2d1 [feature-wip](new-scan) support Json reader (#13546)
Issue Number: close #12574
This pr adds `NewJsonReader` which implements GenericReader interface to support read json format file.

TODO:
1. modify `_scann_eof` later.
2. Rename `NewJsonReader` to `JsonReader` when `JsonReader` is deleted.
2022-10-26 12:52:21 +08:00
44c9163b3c [Fix](multi-catalog)Fix partition external table query bug. (#13535)
The index for external table columns from path is incorrect in new scanner. This is a fix for it.
e.g. In the next query, nation and city columns are from path
```
mysql> select nation, city, count(*) from parquet_two_part group by nation, city;
+--------+------------+----------+
| nation | city       | count(*) |
+--------+------------+----------+
| cn     | beijing    |  1199969 |
| cn     | shanghai   |  1199771 |
| jp     | tokyo      |   599715 |
| rus    | moscow     |   600659 |
| us     | chicago    |  1199805 |
| us     | washington |  1201296 |
+--------+------------+----------+
6 rows in set (0.39 sec)
```
2022-10-26 12:47:37 +08:00
15130c469f [fix](planner) cannot recogonize column's table when analyze rewrite expr (#13597)
We save mv column with alias as table name, and search it with original table name.
2022-10-26 11:15:48 +08:00
e5b33abd3c [fix](planner) inlineView alias error (#13600) 2022-10-26 10:14:04 +08:00
e385cb063c [improvement](config) allow to modify the master-only configuration of non-master nodes (#13558)
Non-master nodes may switch to master, so we should allow the master-only configuration of all fe nodes to be modified.
2022-10-26 10:00:12 +08:00
c709998faa [improvement][refactor](mysql) remove old mysql server and add keep alive option (#13663)
* [improvement][refactor](mysql) remove old mysql server and add keep alive option
2022-10-26 09:38:33 +08:00
295d887cf5 [improvement](thread) set name for priority thread pool (#13552) 2022-10-26 09:32:15 +08:00
a02a56eb38 [fix](postgresql) fix postgresql cann't find table (#13550) 2022-10-26 09:30:28 +08:00
bd2b803019 [opt](tpch) opt the q9 of tpch to change instance to 8 (#13656) 2022-10-26 09:28:51 +08:00
e00734348b [Chore](regression) Fix wrong result for decimal (#13644) 2022-10-26 09:24:46 +08:00
b13061360f [enhancement](chore) build fe-common when build java-udf (#13647)
* [enhancement](chore) Enhance build script compatibility

* remove duplicate fix
2022-10-26 09:23:21 +08:00
c486d9746d [fix](broker) fix bug when broker load with s3a (#13650)
Signed-off-by: nextdreamblue <zxw520blue1@163.com>

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2022-10-26 09:22:38 +08:00
2563dcca95 [fix](load) fix core dump when get_memtable_consumption_inflush (#13629)
If delta writer is not inited, _flush_token might be nullptr.
2022-10-26 09:20:33 +08:00
9691db7918 [Enhancement](metrics) add more metrics (#11693)
* Add `AutoMappedMetric` to measure dynamic object.
* Add query instance and rpc metrics
* Add thrift rpc metrics
* Add txn metrics
* Reorganize metrics init routine.

Co-authored-by: 迟成 <chicheng@meituan.com>
2022-10-26 08:31:03 +08:00
458cb8f072 [chore](macOS) Support macOS Ventura (13.0) (#13626)
We need the tool `makeinfo` to build third parties on macOS Ventura (13.0).
2022-10-26 08:15:21 +08:00
6f18726f01 [improvement](test) add sync for test_agg_keys_schema_change_datev2 (#13643)
1. add "sync" to avoid some potential meta sync problem when running regression test on multi-node cluster
2. Use /tmp dir as dest dir of outfile test, to avoid "No such file or directory" error.
2022-10-25 22:29:05 +08:00
17ba40f947 [feature-wip](CN Node)Support compute node (#13231)
Introduce the node role to doris, and the table creation and tablet scheduler will control the storage only assign to the BE nodes.
2022-10-25 21:44:33 +08:00
cb39671a73 [fix](policy) Add readlock for show policy (#13497)
Add readlock for show policy resolve ConcurrentModificationException
2022-10-25 21:42:40 +08:00
4db18c139c [fix](thirdparty) g++-11: error: unrecognized command-line option '-m… (#13607)
When compiling third-party hdfs on aarch64 machine,
got error: g++-11: error: unrecognized command-line option '-msse4.2'
2022-10-25 21:35:31 +08:00
d6c3470c8d [feature](Nereids) support materialized index selection (#13416)
This PR unified the selection of rollup index and materialized view index into uniform logic, which is called selecting materialized index. 

Main steps:

### Find candidate indexes
1. When aggregate is present, it's handled in `SelectMaterializedIndexWithAggregate`.  The base index and indexes that could use pre-aggregation should be used. The pre-aggregation status is determined by aggregation function, grouping expression, and pushdown predicates.
2. When aggregate is not on top of scan node, it's handled in `SelectMaterializedIndexWithoutAggregate`. The base index and indexes that have all the key columns could be used.

### Filter and order the candidate indexes
1. filter indexes that contain all the required output scan columns.
2. filter indexes that could match prefix index most.
3. order the result index by row count, column count, and index id.
2022-10-25 19:25:58 +08:00
Pxl
bd884d3298 [Chore](build) add a environment variable DISABLE_JAVA_UDF (#13588) 2022-10-25 17:46:56 +08:00
2c70b17a47 [Del](vec) Support in predicate in delete condition of or and (#13587) 2022-10-25 17:33:35 +08:00
1741a20689 [opt](planer) remove unless cast of avg function (#13593) 2022-10-25 17:33:02 +08:00
f209b7ab6e [fix](Nereids) add exchange node check between local and global agg in plan translator (#12913)
### table schema
CREATE TABLE `t1` (
  `k1` int(11) NULL,
  `v1` int(11) NULL
) ENGINE=OLAP
DUPLICATE KEY(`k1`, `v1`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`k1`)
BUCKETS 3
PROPERTIES('replication_num'='1')

### query
select k1,count(distinct v1+1) from t1 group by k1;

### error
java.lang.ClassCastException: org.apache.doris.planner.OlapScanNode cannot be cast to org.apache.doris.planner.AggregationNode
2022-10-25 16:55:29 +08:00
e103531e69 [fix](sort)order by constant expr bug (#13613)
Issue Number: close (#13350)
2022-10-25 16:43:18 +08:00
b85c78ee00 [fix](regression) add 'if not exists' to 'create table' to support parallel test (#13576) (#13578)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2022-10-25 16:37:07 +08:00
78278f5943 [chore](be version) Check BE version by script (#13594)
Check BE version by script
2022-10-25 16:20:38 +08:00
235c105554 [feature-array](array-type) Add array function array_enumerate (#13612)
Add array function array_enumerate
2022-10-25 15:12:11 +08:00
de5bc6a8a5 [fix](regression-test) set label for stream load (#13620) 2022-10-25 14:13:24 +08:00
f802fc37ff add date function 'last_day' (#13609) 2022-10-25 13:46:16 +08:00
57f479d9d2 [fix](test) let cases use their own table name (#13602) 2022-10-25 13:45:01 +08:00
c24e5585c3 [fix](load) clear and notify when an error happens in flushing (#13589) 2022-10-25 13:39:17 +08:00
7fe7c01125 [Bug](decimal) Fix incorrect result for decimal multiply (#13591)
Fix incorrect result for decimal multiply
2022-10-25 12:08:49 +08:00
57e248e09b [feature-wip](unique-key-merge-on-write) check whether the partition column is a key column when create table for MOW table (#13490) 2022-10-24 21:16:38 +08:00
2cf89c55c2 [chore](macOS) Fix issues found on macOS x86_64 (#13583)
1. Use `brew --prefix` instead of `brew --repo` in scripts.
2. `sprintf` is marked as a deprecated function in MacOSX sdk (13.0).
2022-10-24 20:59:20 +08:00
409bd76999 [improve](Nereids): ReorderJoin eliminate this recursion (#13505) 2022-10-24 17:11:43 +08:00
7faad9f004 [FIX](agg)fix group by constant child expr bug (#13485) 2022-10-24 16:32:36 +08:00
40e122e5ef [fix](join)the build and probe expr should be calculated before converting input block to nullable (#13436)
* [fix](join)the build and probe expr should be calculated before converting input block to nullable

* remove_nullable can be called on const column
2022-10-24 14:50:06 +08:00
177e82bdab [Enhancement](array-type) Add type derivation for array functions (#13534)
From now, we don't support type derivation for array function's arguments.
So that the cases below will return wrong values or even cause be core.

mysql> select array_union([1],[10000000]);
+----------------------------------------+
| array_union(ARRAY(1), ARRAY(10000000)) |
+----------------------------------------+
| [1, -128]                              |
+----------------------------------------+
1 row in set (0.03 sec)

mysql> select array_union([NULL],[1]);
ERROR 1105 (HY000): RpcException, msg: io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason

mysql> select array_union([],[1]);
ERROR 1105 (HY000): RpcException, msg: io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
This commit make a small fix to derivate the argument types of the array function
1、 For null type in arguments, cast the null type to boolean type, because null type should not be seen in be.
2、For different types in arguments, cast all arguments type to their compatible type.
2022-10-24 11:51:47 +08:00
54545c6446 [improvement](config) enlarge default value of create_table_timeout and remove disable_stream_load_2pc (#13520)
Users do not need to set create_table_timeout, it is a ddl command and when encounter a timeout event
users will set a lager timeout and retry.

Stream load 2pc is used by default in flink connector, so we should not disable it by config, the config
item is useless.
2022-10-24 11:51:18 +08:00