doris

Files

Ashin Gau 2c8de30cce [optimize](multi-catalog) use dictionary encode&filter to process delete files (#15441 )

**Optimize**
PR #14470 has used `Expr` to filter delete rows to match current data file,
but the rows in the delete file are [sorted by file_path then position](https://iceberg.apache.org/spec/#position-delete-files)
to optimize filtering rows while scanning, so this PR remove `Expr` and use binary search to filter delete rows.

In addition, delete files are likely to be encoded in dictionary, it's time-consuming to decode `file_path`
columns into `ColumnString`, so this PR use `ColumnDictionary` to read `file_path` column.

After testing, the performance of iceberg v2's MOR is improved by 30%+.

**Fix Bug**
Lazy-read-block may not have the filter column, if the whole group is filtered by `Expr`
and the batch_eof is generated from next batch.

2022-12-30 08:57:55 +08:00

check/checkstyle

[feature-wip](array) remove array config and check array nested depth (#13428 )

2022-10-20 15:52:31 +08:00

fe-common

[feature-wip](recover) new recover ddl and support show catalog recycle bin (#13067 )

2022-10-31 17:44:56 +08:00

fe-core

[optimize](multi-catalog) use dictionary encode&filter to process delete files (#15441 )

2022-12-30 08:57:55 +08:00

hive-udf

[deps](fe)upgrade deps version (#15262 )

2022-12-24 22:18:10 +08:00

java-udf

[vectorized](jdbc) support array type in jdbc external table (#15303 )

2022-12-30 00:29:08 +08:00

spark-dpp

[Enhancement](fe): replace assertTrue(X.equals(X)) with assertEquals (#15356 )

2022-12-27 00:37:24 +08:00

pom.xml

[deps](fe)upgrade deps version (#15262 )

2022-12-24 22:18:10 +08:00

README

…

README

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

# fe-common

This module is used to store some common classes of other modules.

# spark-dpp

This module is Spark DPP program, used for Spark Load function.
Depends: fe-common

# fe-core

This module is the main process module of FE.
Depends: fe-common, spark-dpp