Truncate char or varchar columns if size is smaller than file columns or not found in the file column schema by session var `truncate_char_or_varchar_columns`.
This function can be used to replace bitmap_union(to_bitmap(expr))， because bitmap_union(to_bitmap(expr)) need create many many small bitmaps firstly and then merge them into a single bitmap.
bitmap_agg will convert the column value into a bitmap directly. Its performance is better than bitmap_union(to_bitmap(expr)) . In our test , there is about 30% improvement.
Implement the TransientTaskRegister to support submitting transient tasks which do not require a timer trigger.
rename some class:
TimerTaskDisruptor -> TaskDisruptor
TimerTaskEvent -> TaskEvent
TimerTaskExpirationHandler -> TaskHandler
AsyncJobManager -> TimerJobManager
MemoryTask -> TransientTask
make PlanNode.getNumInstance() abstract to force every PlanNode specify how to define its numInstance.
By default, PlanNode.numInstance is 1. PlanNode except exchangeNode should not use this default value directly. PlanNode.numInstance is used for PlanNode which will change numInstance like exchange node.
Execute Sql
delete from test_table.
2023-08-09 11:51:46,586 WARN (mysql-nio-pool-7|540) [StmtExecutor.analyze():987] Analyze failed. stmt[25, 519f916eeb94a8b-afe8e1094fb39fc1]
java.lang.NullPointerException: null
at org.apache.doris.rewrite.ExprRewriter.applyRuleBottomUp(ExprRewriter.java:236) ~[classes/:?]
at org.apache.doris.rewrite.ExprRewriter.applyRule(ExprRewriter.java:226) ~[classes/:?]
at org.apache.doris.rewrite.ExprRewriter.applyRuleRepeatedly(ExprRewriter.java:216) ~[classes/:?]
at org.apache.doris.rewrite.ExprRewriter.rewrite(ExprRewriter.java:166) ~[classes/:?]
at org.apache.doris.rewrite.ExprRewriter.rewrite(ExprRewriter.java:151) ~[classes/:?]
at org.apache.doris.analysis.DeleteStmt.analyze(DeleteStmt.java:127) ~[classes/:?]
at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:983) ~[classes/:?]
at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:660) ~[classes/:?]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:448) ~[classes/:?]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:419) ~[classes/:?]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:441) ~[classes/:?]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:589) ~[classes/:?]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:826) ~[classes/:?]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[classes/:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Fix Result
[HY000][1105] errCode = 2, detailMessage = Where clause is not set
Affected version
2.0-Alpha +
Add scanner isolation class loader to make each plugin non-conflicting.
The BE will get scanner classes by JNI call and use JniClassLoader load them.
In the last version，we always get canner classes from the system class path by default,
so it cannot isolate the classes for each scanner
1. make mv matched when preagg have value column predicate contained in mv
'where clause
2. fix `org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = BITMAP_UNION need input a bitmap column, but input INVALID_TYPE`
3. make the error message more detailed when create mv stmt parse failed
Currently, Doris only has one interface for querying specific FE session information, and many times we need to know how many session information there are in the current cluster, so I added this API.
`
GET /rest/v1/session/all
{
"msg": "success",
"code": 0,
"data": {
"column_names": ["FE", "Id", "User", "Host", "Cluster", "Db", "Command", "Time", "State", "Info"],
"rows": [{
"FE": "10.14.170.23",
"User": "root",
"Command": "Sleep",
"State": "",
"Cluster": "default_cluster",
"Host": "10.81.85.89:31465",
"Time": "230",
"Id": "0",
"Info": "",
"Db": "db1"
},
{
"FE": "10.14.170.24",
"User": "root",
"Command": "Sleep",
"State": "",
"Cluster": "default_cluster",
"Host": "10.81.85.88:61465",
"Time": "460",
"Id": "1",
"Info": "",
"Db": "db1"
}]
},
"count": 2
}
`
The original implement use MutableMap on PhysicalTopN.
It is easy to lose if we rewrite the plan after this processor.
The new implement use attr to indict whether use topn runtime filter
Add Backend tag location check.
Avoid user set a bad backend tag, cause create table and dynamic partitions failed.
For example, the default value for all backends tag is default, When setting the replication_allocation of a table, user use the following command: ALTER TABLE example_db.mysql_table SET ("replication_allocation" = "tag.location.tag1: 1");, it can set success, but tag1 is not exist, cause dynamic partition can't create.
Since we use division calculation, when the start time is not specified,
it may have a wrong deviation from our expected time.
For example, if it is the 7th minute now, the cycle is executed every two minutes.
Then it is calculated that the first execution is 8 minutes Because 7/2=3
3+1=4
But ideally we think it should be executed at the 9th minute
when both decimalv2 and float like type in the arithmetic expr, the common type is depend on roundPreciseDecimalV2Value session variable. If it's true, the common type is DecimalV2Type.SYSTEM_DEFAULT, otherwise its double type.
This PR was originally #16940 , but it has not been updated for a long time due to the original author @Cai-Yao . At present, we will merge some of the code into the master first.
thanks @Cai-Yao @yiguolei
when pushing down constant conjunct into set operation node, we should assign the conjunct to agg node if there is one. This is consistant with pushing constant conjunct into inlineview.
count_by_enum(expr1, expr2, ... , exprN);
Treats the data in a column as an enumeration and counts the number of values in each enumeration. Returns the number of enumerated values for each column, and the number of non-null values versus the number of null values.
