1. Fix value idx in bool rle decoder
2. Iceberg table support datetimev2(3). In the previous version, we converted hive timestamp to datetimev2(0) default.
1. Introduce hadoop libhdfs
2. For Linux-X86 platform, use the hadoop libhdfs
3. For other platform, use libhdfs3, because currently we don't have hadoop libhdfs binary for other platform
Co-authored-by: adonis0147 <adonis0147@gmail.com>
sql like the format (q1, q2, q3 is a query):
``` sql
(q1)
UNION ALL (q2)
UNION ALL (q3)
ORDER BY keys
```
cannot be parsed by nereids, because order will be recognized as an alias of query, we add queryOrganization to avoid it.
In clickhouse's 4.x version of jdbc, some UInt types use special Java types, so I adapted Doris's ClickHouse JDBC External
```
com.clickhouse.data.value.UnsignedByte;
com.clickhouse.data.value.UnsignedInteger;
com.clickhouse.data.value.UnsignedLong;
com.clickhouse.data.value.UnsignedShort;
```
1. add PassNullPredicate to fix topn wrong result for NULL values
2. refactor RuntimePredicate to avoid using TCondition
3. refactor using ordering_exprs in fe and vsort_node
Since slot that reference to constant has been marked as constant expr either, just add condition check to make sure such slot wouldn't be eliminated as constant from group exprs
we introduce this rule by PR #17968, but some corner case do
not be processed correctly. This PR fix these bugs:
1. fix window function generation method, replace inner slot with
equivalent outer slot
2. forbid below scenes
a. inner has a mapping project
b. inner has an unexpected filter
c. outer has a mapping project
d. outer has an unexpected filter
e. outer has additional table
f. outer has same table
g. outer and inner with different join condition
h. outer and inner has same table with different join condition
forbiden create mv with where clause contained aggregate column
create table a_table(
k1 int null,
k2 int not null,
k3 bigint null,
k4 bigint sum null,
k5 bitmap bitmap_union null,
k6 hll hll_union null
)
aggregate key (k1,k2,k3)
distributed BY hash(k1) buckets 3
properties("replication_num" = "1");
create materialized view where_1 as select k1,k4 from a_table where k4 =1; // invalid, mv on agg table need group by
create materialized view where_2 as select k1,sum(k4) from a_table where k4 =1 group by k1; // invalid, k4 is agg column
create materialized view where_2 as select k1,sum(k4) from a_table where k1+k4 =1 group by k1; // invalid, k4 is agg column
nested alias function will cause bind argument exception, sql like:
``` sql
CREATE ALIAS FUNCTION f1(DATETIMEV2(3), INT)
with PARAMETER (datetime1, int1) as date_trunc(days_sub(datetime1, int1), 'day')
CREATE ALIAS FUNCTION f2(DATETIMEV2(3), int)
with PARAMETER (datetime1, int1) as DATE_FORMAT(HOURS_ADD(
date_trunc(datetime1, 'day'),
add(multiply(floor(divide(HOUR(datetime1), divide(24,int1))), 1), 1)
), '%Y%m%d:%H')
select f2(f1(now(3), 2), 3)
```
bug in FunctionCallExpr#rewriteExpr(), the retExpr will be replaced to originExpr to change the alias function to builtin function, but the retExpr.fn is not null, so when return to outer scope, the fn will be covered. That's the example:
```
f1(f1()) -> date_trunc(days_sub(date_trunc(days_sub()))) is correct and
f1(f1()) -> date_trunc(days_sub(days_sub())) is bug.
```
we fix it.
1. add LateralViewRef's id into TableRef's allTableRefIds, so the caller won't miss LateralViewRef when trying to get all the tableref ids.
2. TableFunctionNode should use child node's output tuple id as the input tuple id
improve column match performance by introducing a column name map in `MaterializedIndexMeta`
`getColumnByName` is slow due to the linear search process, using a map to speed up search.
avoid warn log in specializeTemplateFunction at initialization like this:
```
2023-03-23 21:17:54,931 INFO (leaderCheckpointer|89) [FunctionSet.specializeTemplateFunction():1337] specializeTemplateFunction exception at initialize
org.apache.doris.catalog.TypeException: ARRAY<DECIMALV3(9, 0)> is not MapType
at org.apache.doris.catalog.MapType.specializeTemplateType(MapType.java:137) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.FunctionSet.specializeTemplateFunction(FunctionSet.java:1321) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.FunctionSet.getFunction(FunctionSet.java:1251) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.FunctionSet.getFunction(FunctionSet.java:1216) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.FunctionSet.addBuiltinBothScalaAndVectorized(FunctionSet.java:1449) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.FunctionSet.addScalarAndVectorizedBuiltin(FunctionSet.java:1432) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.builtins.ScalarBuiltins.initBuiltins(ScalarBuiltins.java:108) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.FunctionSet.init(FunctionSet.java:87) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.<init>(Env.java:585) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.getCurrentEnv(Env.java:665) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.master.Checkpoint.doCheckpoint(Checkpoint.java:143) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.master.Checkpoint.runAfterCatalogReady(Checkpoint.java:77) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
```
* Revert "[fix](function) fix AES/SM3/SM4 encrypt/ decrypt algorithm initialization vector bug (#17420)"
This reverts commit 397cc011c4f1ba5a25c770258c13f1cd3f28b47d.
* [fix-resubmit](function) fix AES/SM3/SM4 encrypt/ decrypt algorithm initialization vector bug (#17420)
ECB algorithm, block_encryption_mode does not take effect, it only takes effect when init vector is provided.
Solved: 192/256 supports calculation without init vector
For other algorithms, an error should be reported when there is no init vector
Initialization Vector. The default value for the block_encryption_mode system variable is aes-128-ecb, or ECB mode, which does not require an initialization vector. The alternative permitted block encryption modes CBC, CFB1, CFB8, CFB128, and OFB all require an initialization vector.
Reference: https://dev.mysql.com/doc/refman/8.0/en/encryption-functions.html#function_aes-decrypt
Note: This fix does not support smooth upgrades. during upgrade process, query may report error: funciton not found
Fix two problems,
1. The logical join containing the MarkJoinSlotRefrance column will generate a plan->MarkJoinSlotreference structure when reorderJoin is executed, and the MarkJoinSlotreference column will be restored after the reorder is completed. But when filter+crossJoin exists, it will be transformed into innerJoin in the rules, causing the map to fail, and the corresponding plan cannot be found, thus losing the MarkJoinSlotreference column.
2. Originally, the MarkJoinSlotReference column was used as the NonUserVisibleOutput of logicalJoin. At the same time, when logicalApply was generated, the added logicalProject did not include the MarkJoinSlotReference column, and the invalid logicalProject was deleted based on other rules, so as to ensure that LogicalApply was under the logicalFilter and could recognize the MarkJoinSlotReference column. But there will be problems if logicalProject cannot be deleted.
Repair method
1. For logicalJoin containing MarkJoinSlotreference, the rules of reorderJoin are not executed.
2. Use MarkJoinSlotreference as the output of logicalJoin and also as the output of LogicalApply.
3. When generating LogicalApply, if MarkJoinSlotreference is included, you need to add an additional logicalProject to logicalFilter, and remove the MarkJoinSlotreference column.
eg
```
logicalFilter(subquery with disconjunct)
after SubqueryToApply
logicalProject(without markJoinSlotReference)
+-- logicalFilter(markJoinSlotReference)
+-- logicalProject(with markJoinSlotReference)
+-- logicalApply()
```
```
SELECT * FROM sub_query_correlated_subquery1 WHERE k1 IN (SELECT k1 FROM sub_query_correlated_subquery3) OR k1 < 10;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| LogicalProject[60] ( distinct=false, projects=[k1#0, k2#1], excepts=[], canEliminate=true ) |
| +--LogicalProject[59] ( distinct=false, projects=[k1#0, k2#1], excepts=[], canEliminate=true ) |
| +--LogicalFilter[58] ( predicates=($c$1#7#false OR (k1#0 < 10)) ) |
| +--LogicalProject[57] ( distinct=false, projects=[k1#0, k2#1, $c$1#7#false], excepts=[], canEliminate=true ) |
| +--LogicalApply ( correlationSlot=[], correlationFilter=Optional.empty, isMarkJoin=true, MarkJoinSlotReference=$c$1#7#false, scalarSubCorrespondingSlot=empty ) |
| |--LogicalOlapScan ( qualified=default_cluster:regression_test_nereids_syntax_p0.sub_query_correlated_subquery1, indexName=<index_not_selected>, selectedIndexId=63105, preAgg=ON ) |
| +--LogicalProject[34] ( distinct=false, projects=[k1#2], excepts=[], canEliminate=true ) |
| +--LogicalOlapScan ( qualified=default_cluster:regression_test_nereids_syntax_p0.sub_query_correlated_subquery3, indexName=<index_not_selected>, selectedIndexId=63115, preAgg=ON ) |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
1. separate the planning process for legacy planner and Nereids in StmtExecutor
2. add forward to master logic to Nereids
3. refactor Command process for Nereids, add run interface to Command
4. internal query could run on Nereids as normal query
5. fix CreatePolicyCommand syntax, let it exactlly same with legacy planner
6. let Nereids session variables forward to master
Follow #17586.
This PR mainly changes:
Remove env/
Remove FileUtils/FilesystemUtils
Some methods are moved to LocalFileSystem
Remove olap/file_cache
Add s3 client cache for s3 file system
In my test, the time of open s3 file can be reduced significantly
Fix cold/hot separation bug for s3 fs.
This is the last PR of #17764.
After this, all IO operation should be in io/fs.
Except for tests in #17586, I also tested some case related to fs io:
clone
concurrency query on local/s3/hdfs
load error log create and clean
disk metrics
Found that RPC timeout is too short that RPC client will close before execute result is return.
Therefore, use a coefficient to prolong the RPC client timeout, so that it can wait for the real cause to be recieved.
