```
2023-11-05 22:52:01,914 WARN (mysql-nio-pool-167|1655) [StmtExecutor.analyze():992] Analyze failed. stmt[222101, ec1adf0f9f7b405d-8cb71a522bc50b82]
java.lang.NullPointerException: null
at org.apache.doris.planner.OlapScanNode.computeSampleTabletIds(OlapScanNode.java:952) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.OlapScanNode.init(OlapScanNode.java:548) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.SingleNodePlanner.createScanNode(SingleNodePlanner.java:2064) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:2213) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:1244) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:266) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:189) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.OriginalPlanner.createPlanFragments(OriginalPlanner.java:160) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.planner.OriginalPlanner.plan(OriginalPlanner.java:101) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.analyzeAndGenerateQueryPlan(StmtExecutor.java:1141) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:975) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:673) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:451) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:422) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:435) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:583) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:834) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_333]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_333]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_333]
2023-11-05 22:52:01,914 WARN (mysql-nio-pool-167|1655) [StmtExecutor.executeByLegacy():776] execute Exception. stmt[222101, ec1adf0f9f7b405d-8cb71a522bc50b82]
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Unexpected exception: null
at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:993) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:673) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:451) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:422) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:435) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:583) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:834) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_333]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_333]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_333]
```
1. prune rf when column stats are not available
2. print rf in "explain shape plan", both join side and apply side
3. add regression case to check plan shape/rf/rf prune for tpch_sf1000 (stats are not available)
This PR adds the processing of lowercase Column names in Oracle Jdbc Catalog. In the previous behavior, we changed all Oracle columns to uppercase queries by default, but could not handle the lowercase case. This PR can solve this situation and improve All Jdbc Catalog works
When a table is doing schema-change, it adds _doris_shadow prefix in name of modified columns in shadow index.
The writes during schema-change will generate rowset schema with _doris_shadow prefix in BE.
If the alter task arrives at be after the write request, it will use the rowset schema with max version which has the _doris_shadow prefix.
And an error will be thrown as below:
a shadow column is encountered __doris_shadow_p_retailprice
[INTERNAL_ERROR]failed due to operate on shadow column
This commit will disable shadow prefix in rowset meta schema.
1. Do not use FATAL log when jni encounter error, to avoid crash.
2. Fix NPE when closing PaimonReader, the reader may not be assigned if PaimonReader open failed.
occasional failure of fe ut, borrowObject throw npe
```
get agent task request. type: CREATE, signature: 10008, fe addr: null
java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.register(GenericKeyedObjectPool.java:1079)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:347)
get agent task request. type: CREATE, signature: 10012, fe addr: TNetworkAddress(hostname:127.0.0.1, port:56072)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277)
at org.apache.doris.common.GenericPool.borrowObject(GenericPool.java:99)
at org.apache.doris.utframe.MockedBackendFactory$DefaultBeThriftServiceImpl$1.run(MockedBackendFactory.java:219)
at java.lang.Thread.run(Thread.java:750)
```
Fix two bugs:
1. Missing column is case sensitive, change the column name to lower case in FE for hive/iceberg/hudi
2. Iceberg use custom method to encode special characters in column name. Decode the column name to match the right column in parquet reader.
Since #24830 introduce `broker.name` in hms catalog, data scan will run on specified brokers.
And [doris operator](https://github.com/selectdb/doris-operator) support BE and broker deployed in same pod, BE access local broker is the fastest approach to access data.
In previous logic, every inputSplit will select one BE to execute, then randomly select one broker for actual data access, BE and related broker are always located on separate K8S pod.
This pr optimizes the broker select strategy to prioritize BE-local broker when `broker.name` is specified in hms catalog.
This pull request addresses an issue observed with inverted index tables or tables without indices when querying null values using the MATCH function.
Previously, executing a query like `SELECT * FROM table WHERE column MATCH null;` would yield incorrect results.
The update introduces enhanced handling of nullable columns within the MATCH function, ensuring accurate query results when null values are involved.
- Fix complex type crash when using the dict filter facility in the parquet-reader by turning off the dict filter facility in this case.
- Add orc complex types regression test.
