Commit Graph

8276 Commits

Author SHA1 Message Date
1dabcb0111 [Fix](Nereids) fix except and intersect error for statsCalculator (#15557)
When calculating the statsCalculator of except and intersect, the slotId of the corresponding column was not replaced with the slotId of output, resulting in NPE.
2023-01-03 17:06:57 +08:00
caaae28b50 [docs](export) fix export data to object storage docs (#15563) 2023-01-03 16:45:02 +08:00
8a5f1351e2 [typo](doc) add be max jvm heap size config description (#15561) 2023-01-03 16:04:18 +08:00
b50448d5c4 [vectorized](udaf) fix udaf result is null when has multiple aggs (#15554) 2023-01-03 16:03:43 +08:00
8748f65a1b [fix](nereids)support nulls first/last in order by clause (#15530) 2023-01-03 14:56:00 +08:00
893f5f9345 [feature-wip](multi-catalog) support automatic sync hive metastore events (#15401)
Poll metastore for create/alter/drop operations on database, table, partition events at a given frequency.
By observing such events, we can take appropriate action on the (refresh/invalidate/add/remove)
so that represents the latest information available in metastore.
We keep track of the last synced event id in each polling
iteration so the next batch can be requested appropriately.
2023-01-03 13:59:14 +08:00
9ab663212b [docs](muti-catalog) update external iceberg system doc (#15556)
Tell users how to solve the problem "fail to read schema from table xx or Storage schema reading not supported"
when doris access hive metastore.
2023-01-03 13:58:16 +08:00
77fda4f749 [SpillToDisk](block reader and writer)Support spill to disk: implement interfaces for spill block and read block (#15399) 2023-01-03 12:42:45 +08:00
ada72b055f [feature](Nereids): Support any_value/any function. (#15450) 2023-01-03 12:21:13 +08:00
02d035466b [refactor] remove partition pruner v1 (#15552)
partition pruner v1 is no longer used.
Also remove session variable partition_prune_algorithm_version
2023-01-03 11:35:30 +08:00
31548cfe2a [fix](nereids) check failed that exchange node under agg must from PhysicalDistribute (#15473)
when nereids translates PhysicalHashAggreg node to original plan, if the input fragment root is exchange node, nereids assumes that this exchanged node is generated from PhyscialDistirbute node.
But this assumption is not true. For example, sort node could be translated to exchange(merge phase)+sort(local phase).
2023-01-03 11:19:25 +08:00
57620f6f0d [bugfix](datetimev2) fix coredump when load datatime data to doris (#15510) 2023-01-03 10:05:44 +08:00
17286861ef [Fix](multi catalog)Skip non-vectorized init code for NewFileScanNode. #15550 2023-01-03 09:22:17 +08:00
14eaf41029 [refactor](remove rowblockv2) remove rowblock v2 structure (#15540)
* [refactor](remove rowblockv2) remove rowblock v2 structure

* fix bugs

Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-01-03 09:21:57 +08:00
40c53931e5 [fix](vec) VMergeIterator add key same label for agg table (#14722) 2023-01-02 22:54:21 +08:00
365c3eec16 [enhancement](compaction) vertical compaction support unique-key mow (#15353) 2023-01-02 22:53:04 +08:00
5062c62ee1 [chore](script) add build-for-release.sh (#15545) 2023-01-02 22:50:36 +08:00
87110ad3e3 [chore](Sink)remove useless OlapTablePartitionParam-related code (#15549) 2023-01-02 22:47:16 +08:00
5d145cf86f [fix](regression-test) fix duplicate columns in yandex_metrica_p2 case (#15489) 2023-01-02 20:31:46 +08:00
238ae54620 [fix](merge-on-write) unique key mow tables should require distribution columns be key column (#15535)
* [fix](merge-on-write) unique key mow tables should require distribution columns be key column

* fix code style
2023-01-01 15:53:21 +08:00
ad9a67a76a [Bug](decimalv3) Fix wrong decimalv3 value after insertion (#15505) 2023-01-01 11:08:59 +08:00
7a433a19e6 [fix](typo) Fix typo in variable name (#15538) 2023-01-01 11:04:08 +08:00
3c2dee1d10 [fix](typo) Fix typo in variable name (#15538) 2023-01-01 11:03:45 +08:00
62f43c3314 [doc](thirdparty) update CHANGELOG, add libhdfs3 2.3.4 (#15539) 2023-01-01 11:03:05 +08:00
487d159a3d [improvement](test) add one case for hll (#15543) 2023-01-01 11:02:34 +08:00
50f1931f96 [fix](multi-catalog) get dictionary-encode from parquet metadata (#15525) 2022-12-31 19:08:10 +08:00
e89adc6e1d [fix](create-table) wrong judgement about partition column type (#15542)
The following stmt should be success, but return error: `complex type cannt be partition column:ARRAY<VARCHAR(64)>`

```
create table test_array( 
task_insert_time BIGINT NOT NULL DEFAULT "0" COMMENT "" , 
task_project ARRAY<VARCHAR(64)>  DEFAULT NULL COMMENT "" ,
route_key DATEV2 NOT NULL COMMENT "range分区键"
) 
DUPLICATE KEY(`task_insert_time`)  
 COMMENT ""
PARTITION BY RANGE(route_key) 
(PARTITION `p202209` VALUES LESS THAN ("2022-10-01"),
PARTITION `p202210` VALUES LESS THAN ("2022-11-01"),
PARTITION `p202211` VALUES LESS THAN ("2022-12-01")) 
DISTRIBUTED BY HASH(`task_insert_time` ) BUCKETS 32 
PROPERTIES
(
    "replication_num" = "1",    
    "light_schema_change" = "true"    
);
```

This PR fix this
2022-12-31 13:10:39 +08:00
c47bdf6606 [vectorized](jdbc) fix external table of oracle have keyworld column (#15487)
if column name is keyword of oracle, the query will report error
2022-12-31 12:48:26 +08:00
781fa17993 [fix](Nereids) round function return type should be double (#15502) 2022-12-30 23:36:15 +08:00
96518db263 [enhencement](Nereids) remove constant expr constraint on OneRowRelation (#15506) 2022-12-30 23:35:15 +08:00
100834df8b [fix](nereids) fix some arrgregate bugs in Nereids (#15326)
1. the agg function without distinct keyword should be a "merge" funcion in threePhaseAggregateWithDistinct
2. use aggregateParam.aggMode.consumeAggregateBuffer instead of aggregateParam.aggPhase.isGlobal() to indicate if a agg function is a "merge" function
3. add an AvgDistinctToSumDivCount rule to support avg(distinct xxx) in some case
4. AggregateExpression's nullable method should call inner function's nullable method.
5. add a bind slot rule to bind pattern "logicalSort(logicalHaving(logicalProject()))"
6. don't remove project node in PhysicalPlanTranslator
7. add a cast to bigint expr when count( distinct datelike type )
8. fallback to old optimizer if bitmap runtime filter is enabled.
9. fix exchange node mem leak
2022-12-30 23:07:37 +08:00
cc7a9d92ad [refactor](non-vec) remove non vec code for indexed column reader (#15409) 2022-12-30 23:01:54 +08:00
9bba2f4cde [typo](docs) array function doc fix (#15449) 2022-12-30 23:00:48 +08:00
9c3c9db49b [enhancement](fuzzy test) support fuzzy test of RewriteOrToInPredicateThreshold #15469
Co-authored-by: wuhangze <wuhangze@jd.com>
2022-12-30 22:59:59 +08:00
ad68764977 [enhancement](tablet) Unify redundant create_rowset_writer methods (#15519)
* Remove redundant create_rowset_writer methods

* Set resource id when setting FS in rowset meta

* fix

* fix ut
2022-12-30 22:57:12 +08:00
edecc2e706 [feature-wip](inverted index) API for inverted index reader and syntax for fulltext match (#14211)
* [feature-wip](inverted index)inverted index api: reader

* [feature-wip](inverted index) Fulltext query syntax with MATCH/MATCH_ALL/MATCH_ALL

* [feature-wip](inverted index) Adapt to index meta

* [enhance] add more metrics

* [enhance] add fulltext match query check for column type and index parser

* [feature-wip](inverted index) Support apply inverted index in compound predicate which except leaf node of and node
2022-12-30 21:48:14 +08:00
b23d068281 [refactor](remove-non-vec) Remove non vec load from memtable and delta writer (#15517)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2022-12-30 21:22:58 +08:00
aacd11336a [typo](docs)update java udf demo (#15521) 2022-12-30 21:12:34 +08:00
aeaa319203 [fix](fe)change session variable group_concat_max_len from int to long (#15515) 2022-12-30 20:45:44 +08:00
ec52907b06 [fix](index) fix wrong dcheck in indexed column writer (#15520) 2022-12-30 20:12:41 +08:00
8e58d92e77 [typo](docs) fix document info missing in SHOW-TABLETS.md (#15488) 2022-12-30 18:39:21 +08:00
084eec87ee [docs](docs)update en docs (#15470)
* Update basic-summary.md
2022-12-30 18:38:26 +08:00
a7895ba169 [feature](Nereids): Support variance_samp function. (#15500) 2022-12-30 17:32:06 +08:00
34d7eeb571 [doc](session variable) add doc content for adding variables called rewrite_or_to_in_predicate_threshold (#15513)
Co-authored-by: wuhangze <wuhangze@jd.com>
2022-12-30 17:11:45 +08:00
93a25e1af5 [fix](nereids) the project node is lost when creating PhysicalStorageLayerAggregate node (#15467) 2022-12-30 16:33:24 +08:00
08d4dcefff [typo](doc)data partition doc including en and zh-CN #15379
Co-authored-by: Chen Jinquan 陈金泉 (690) <chenjinq@haier.com>
2022-12-30 15:38:25 +08:00
dec1eb360c [fix](brokerload) be core dump caused by broker load orc format file nullptr pointer (#15460) 2022-12-30 15:37:33 +08:00
2f572ccc43 [fix](index) fix that the last element of each batch will be read repeatedly for binary prefix page (#15481) 2022-12-30 15:36:55 +08:00
9246e03932 [Enhancement](hdfs) make libhdfs3 compatible with hdfs2 server (#15497)
When doris be getFileStatus from HDFS2 server, libhdfs3 will throw exception because of the permission code returned by hdfs2 server is greater than 1<<12.
The bit 12 of permission code is aclBit which has been deprecated in hadoop3. so we remove the check code in libhdfs3, same as hadoop3 java project.
2022-12-30 15:36:39 +08:00
2704651fde [fix](nereids) hll and bitmap type can't be used as order by and group by exprs (#15471)
hll, bitmap, array and quantile state type can't be used in order by, group by and some agg exprs.
2022-12-30 14:26:21 +08:00