if broker detail contails ak/sk or some sensitive information, not print these info default in log
---------
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
This reverts commit 5b641ebd40fff71e632ee9be4ede58b744b602b9.
Currently, Deltalake Catalog is not a usable feature. We will continue to implement it in the datalake plug-in system in the future, so we will delete it from the FE code for now.
Single key column of unique/agg olap table has the same value of count and ndv, for this kind of column,
don't need to calculate ndv, simply use count as ndv.
using blocksproduced and rowsproduced to unify the counter name in DataStreamSender and other exec node, or exchange operator and other operators.
blocks produced and rows produced are more easy to understand.
---------
Co-authored-by: yiguolei <yiguolei@gmail.com>
```
2023-11-05 22:52:01,914 WARN (mysql-nio-pool-167|1655) [StmtExecutor.analyze():992] Analyze failed. stmt[222101, ec1adf0f9f7b405d-8cb71a522bc50b82]
java.lang.NullPointerException: null
at org.apache.doris.planner.OlapScanNode.computeSampleTabletIds(OlapScanNode.java:952) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.OlapScanNode.init(OlapScanNode.java:548) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.SingleNodePlanner.createScanNode(SingleNodePlanner.java:2064) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:2213) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:1244) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:266) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:189) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.OriginalPlanner.createPlanFragments(OriginalPlanner.java:160) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.OriginalPlanner.plan(OriginalPlanner.java:101) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.analyzeAndGenerateQueryPlan(StmtExecutor.java:1141) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:975) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:673) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:451) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:422) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:435) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:583) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:834) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_333]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_333]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_333]
2023-11-05 22:52:01,914 WARN (mysql-nio-pool-167|1655) [StmtExecutor.executeByLegacy():776] execute Exception. stmt[222101, ec1adf0f9f7b405d-8cb71a522bc50b82]
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Unexpected exception: null
at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:993) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:673) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:451) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:422) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:435) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:583) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:834) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_333]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_333]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_333]
```
1. prune rf when column stats are not available
2. print rf in "explain shape plan", both join side and apply side
3. add regression case to check plan shape/rf/rf prune for tpch_sf1000 (stats are not available)
This PR adds the processing of lowercase Column names in Oracle Jdbc Catalog. In the previous behavior, we changed all Oracle columns to uppercase queries by default, but could not handle the lowercase case. This PR can solve this situation and improve All Jdbc Catalog works
When a table is doing schema-change, it adds _doris_shadow prefix in name of modified columns in shadow index.
The writes during schema-change will generate rowset schema with _doris_shadow prefix in BE.
If the alter task arrives at be after the write request, it will use the rowset schema with max version which has the _doris_shadow prefix.
And an error will be thrown as below:
a shadow column is encountered __doris_shadow_p_retailprice
[INTERNAL_ERROR]failed due to operate on shadow column
This commit will disable shadow prefix in rowset meta schema.
1. Do not use FATAL log when jni encounter error, to avoid crash.
2. Fix NPE when closing PaimonReader, the reader may not be assigned if PaimonReader open failed.
occasional failure of fe ut, borrowObject throw npe
```
get agent task request. type: CREATE, signature: 10008, fe addr: null
java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.register(GenericKeyedObjectPool.java:1079)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:347)
get agent task request. type: CREATE, signature: 10012, fe addr: TNetworkAddress(hostname:127.0.0.1, port:56072)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277)
at org.apache.doris.common.GenericPool.borrowObject(GenericPool.java:99)
at org.apache.doris.utframe.MockedBackendFactory$DefaultBeThriftServiceImpl$1.run(MockedBackendFactory.java:219)
at java.lang.Thread.run(Thread.java:750)
```
Fix two bugs:
1. Missing column is case sensitive, change the column name to lower case in FE for hive/iceberg/hudi
2. Iceberg use custom method to encode special characters in column name. Decode the column name to match the right column in parquet reader.
Since #24830 introduce `broker.name` in hms catalog, data scan will run on specified brokers.
And [doris operator](https://github.com/selectdb/doris-operator) support BE and broker deployed in same pod, BE access local broker is the fastest approach to access data.
In previous logic, every inputSplit will select one BE to execute, then randomly select one broker for actual data access, BE and related broker are always located on separate K8S pod.
This pr optimizes the broker select strategy to prioritize BE-local broker when `broker.name` is specified in hms catalog.
