When using an equal predicate on a column that applies an inverted index
with a parser, it requires remaining_after_evaluate. In this situation,
we cannot optimize the column without reading the data.
## Proposed changes
From (#36637)
this pr
1. picked #35630, which was reverted #36098 before.
2. picked #36344 from master
these two pr fixed existing bug about auto partition load.
---------
Co-authored-by: Kaijie Chen <ckj@apache.org>
pick from master #36316
expression cast( xx as decimal )'s datatype maybe decimalv3 or decimalv2
depending on enable_decimal_conversion value in fe conf file. if
enable_decimal_conversion is true, the datatype is decimalv3(9, 0), but
the datatype was decimalv3(38, 9) in 2.0 releases. So this pr change the
datatype same as 2.0 releases to keep the behavior consistent.
bp: #36432
## Proposed changes
## Fixed Bugs introduced from #34307
1. `FileSystemCacheKey.equals()` compares properties by `==`, resulting
in creating new file system in each partition
2. `dfsFileSystem` is not synchronized, resulting in creating more file
systems than need.
3. `jobConf.iterator()` will produce more than 2000 pairs of key-value
bp #36045, and turn on batch split, which is turn off in #36109
Generate and get split batch concurrently.
`SplitSource.getNextBatch` remove the synchronization, and make each get their splits concurrently, and `SplitAssignment` generates splits asynchronously.
pick from master #35773
This PR introduces an optimization that adjusts the penalty applied
during join operations based on the volume of data on the build side.
Specifically, when the number of rows and width of the tables being
joined are equal, the materialization costs are now considered more
accurately. The update ensures that joins with a larger dataset on the
build side incur a higher penalty, improving overall query performance
and resource allocation.
cherry-pick #36193
Problem:
when using leading like:
leading(t1 {t2 t3} {t4 t5} t6)
it would not generate correct plan because levellist can not express
enough message of braces
Solved:
remove levellist express of leading levels and use reverse polish
expression
Algorithm:
leading(t1 {t2 t3} {t4 t5} t6)
==>
stack top to down(t1 t2 t3 join join t4 t5 join t6 join) when generate
leading join, we can pop items in stack, when it's a table, make
logicalscan when it's a join
operator, make logical join and push back to stack
cherry pick from #36326
An exception throw in TryEliminateUninterestedPredicates, for this case
CREATE TABLE `tbltest` (
`id` INT NULL,
`col2` VARCHAR(255) NULL,
`col3` VARCHAR(255) NULL,
`dt` DATE NULL
) ENGINE=OLAP
DUPLICATE KEY(`id`, `col2`)
PARTITION BY RANGE(`dt`)
(PARTITION p20240617 VALUES [('2024-06-17'), ('2024-06-18')))
DISTRIBUTED BY HASH(`id`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);
select * from tbltest
where
case
when col2 = 'xxx' and col3='yyy' then false -- note this is not about partition column
when col2 in ('xxx') then false
when col2 like 'xxx%' then false
else true
end
The `CaseWhen` require children should be `WhenClause`, TryEliminateUninterestedPredicates maybe rewrite the WhenClause to true/false predicate, and cause this exception:
ERROR 1105 (HY000): errCode = 2, detailMessage = The children format needs to be [WhenClause+, DefaultValue?]
Original extractor(TryEliminateUninterestedPredicates.java) caused some errors while try to derive the expressions which can be used for pruning partitions.
I tried to write a new extractor(and with unit tests) for pruning partitions, it is more simple and reliable (I think).
The theory of extractor is pretty simple:
A:Sort the expression in two kinds:
1. evaluable-expression (let's mark it as E).
Expressions that can be evaluated in the partition pruning stage.
In the other word: not contains non-partition slots or deterministic expression.
2. un-evaluable-expression (let's mark it as UE).
Expressions that can NOT be evaluated in the partition pruning stage.
In the other word: contains non-partition slots or deterministic expression.
B: Travel the predicate, only point on AND and OR operator, following the rule:
(E and UE) -> (E and TRUE) -> E
(UE and UE) -> TRUE
(E and E) -> (E and E)
(E or UE) -> TRUE
(UE or UE) -> TRUE
(E or E) -> (E or E)
pick from master #35873
Update tpcds tools table customer_demographics's bucket column as its
primary key column, avoid performance issue due to data skew.
