consider the query like this:
```sql
SELECT
k3, k4
FROM
test
WHERE
EXISTS( SELECT
d.*
FROM
(SELECT
k1 AS _1234, SUM(k2)
FROM
`test` d
GROUP BY _1234) d
LEFT JOIN
(SELECT
k1 AS _1234,
SUM(k2)
FROM
`test`
GROUP BY _1234) temp ON d._1234 = temp._1234)
ORDER BY k3, k4
```
when we analyze group by exprs in `temp` inline view. we bind the `_1234` on `d._1234` by mistake.
that because, when we do analyze in a **SUB-QUERY**, we will resolve SlotRef by itself **AND** parent's tuple. in the meanwhile, we register child's tuple to parent's analyzer. So, in a **SUB-QUERY**, the brother's tuple will affect the resolve result of current inlineview's slot.
This PR:
1. add a flag on the function `resolveColumnRef` in `Analyzer`
```java
private TupleDescriptor resolveColumnRef(String colName, boolean requestFromChild);
private TupleDescriptor resolveColumnRef(TableName tblName, String colName, boolean requestByChild);
```
2. add a flag to specify whether the tuple is from child.
```java
// alias name -> <from child, tupleDesc>
private final Multimap<String, Pair<Boolean, TupleDescriptor>> tupleByAlias;
```
when `requestByChild == true`, we **SKIP** the tuple from other child to avoid resolve error.
1. should always execute projection plan, whatever the statement it is.
2. should always execute projection plan, since we only have vectorized engine now
1.When querying data, it is no longer necessary to verify the permissions of the entire table, but rather to verify the
permissions of the queried columns. Currently, the 'ranger' already supports column permissions, and the internal
catalog provides the implementation of dummy column permissions (the actual verified permissions are still table
permissions)
2.delete roles in userIdentity
3.Change trigger logic of initAccessController
See #17764 for details
I have tested:
- Unit test for local/s3/hdfs/broker file system: be/test/io/fs/file_system_test.cpp
- Outfile to local/s3/hdfs/broker.
- Load from local/s3/hdfs/broker.
- Query file on local/s3/hdfs/broker file system, with table value function and catalog.
- Backup/Restore with local/s3/hdfs/broker file system
Not test:
- cold & host data separation case.
Complete the type coercion of the subquery in the function Binder process.
Expressions generated when subqueries are nested are uniformly converted to implicit types in the analyze stage.
Method: Add a typeCoercionExpr field to the subquery expression to store the generated cast information.
Fix scenario where scalarSubQuery handles arithmetic expressions when implicitly converting types
When setting FE config default_storage_medium to SSD, and set all BE storage path as SSD.
And table will be stored with storage medium SSD.
But there is a FE config storage_cooldown_second and its default value is 30 days.
So after 30 days, the storage medium of table will be changed to HDD, which is unexpected.
This PR removes the storage_cooldown_second, and use a max value to set the cooldown time of SSD
storage medium when the default_storage_medium is SSD.
Missing userinfo
java.lang.NullPointerException: null
at org.apache.doris.load.loadv2.LoadJob.getShowInfo(LoadJob.java:816) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.load.loadv2.LoadManager.getLoadJobInfosByDb(LoadManager.java:557) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ShowExecutor.handleShowLoad(ShowExecutor.java:1094) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ShowExecutor.execute(ShowExecutor.java:280) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.handleShow(StmtExecutor.java:1862) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:619) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:435) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:414) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:558) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:799) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_131]
Describe your changes.
1.Add the watch mechanism to listen for changes in k8s statefulSet and update nodes in time.
2.For broker, there is only one name by default when using deployManager
3.Refactoring code makes it easier to understand and maintain
4.Fix jar package conflicts between okhttp-ws and okhttp
Previously, the logic of k8sDeployManager.getGroupHostInfos was to call the endpoints () interface of k8s,
which would cause if the pod was unexpectedly restarted, k8sDeployManager would delete the pod before the
restart from the fe or be list and add the pod after the restart to the fe or be list, which obviously does not
meet our expectations.
Now, after fqdn is enabled, we call the statefulSets() interface of k8s to listen for the number of copies to
determine whether we need to be online or offline.
In addition, the watch mechanism is added to avoid the possible A-B-A problem caused by timed polling.
For the sake of stability, when the watch mechanism does not receive messages for a period of time,
it will be degraded to the polling mode.
Now several environment variables have been added，ENV_FE_STATEFULSET，ENV_FE_OBSERVER_STATEFULSET，ENV_BE_STATEFULSET，ENV_BROKER_STATEFULSET，ENV_CN_STATEFULSET For statefulsetName，One-to-one correspondence with ENV_FE_SERVICE，ENV_FE_OBSERVER_SERVICE，ENV_BE_SERVICE，ENV_BROKER_SERVICE，ENV_CN_SERVICE，If a serviceName is configured, the corresponding statefulsetName must be configured, otherwise the program cannot be started.
This problem is caused by the slots with same hashcodes was put in the hashset results into the wrong rules was selected.Use list instead of set as return type of getDistinctArguments method
remove the redandunt lock in the CacheLoader, since it use the forkjoinpool in default
Add execute time log for collect stats
Avoid submit duplicate task, when there already has a task to load for the same column
In #17797 , we introduced aspectj to help log exception easily.
However, the plugin version 1.11 do not support jdk9 and later.
For support compile FE with jdk11
update aspectj-maven-plugin to 1.14.0 version
add new dependency org.aspectj.aspectjrt 1.9.7 to fe-core
according to:
aspectj java version compatibility
aspectj-maven-plugin issue
aspectj release note
intro to aspectj
Support create/drop global function.
When you create a custom function, it can only be used within in one database. It cannot be used in other database/catalog. When there are many databases/catalog, it needs to create function one by one.
## Problem summary
Describe your changes.
1、 When a function is created or deleted, add the global keyword.
CREATE [GLOBAL] [AGGREGATE] [ALIAS] FUNCTION function_name (arg_type [, ...]) [RETURNS ret_type] [INTERMEDIATE inter_type] [WITH PARAMETER(param [,...]) AS origin_function] [PROPERTIES ("key" = "value" [, ...]) ]
DROP [GLOBAL] FUNCTION function_name (arg_type [, ...])
2、A completely global global function is set, and the global function metadata is stored in the image. The function lookup strategy is to look in the database first, and if it can't be found, it looks in the global function.
Co-authored-by: lexluo <lexluo@tencent.com>
* 1. update ndv in Stats,
2. skip __DORIS_DELETE_SIGN__=0 in stats derive,
3. equalTo in stats derive
4. update agg stats derive, support the case: all column_stats are unknown
* computeSize
* fix ut
basic functions for map datatype:
- MAP<K, V> map(K k1, V v1, ...)
- BIGINT map_size(MAP<K, V> m)
- BOOL map_contains_key(MAP<K, V> m, K k1)
- BOOL map_contains_value(MAP<K, V> m, V v1)
- ARRAY< K> map_keys(MAP<K, V> m)
- ARRAY< V> map_values(MAP<K, V> m)
How it works?
Aspectj is used to implement the aspect function of annotations. During the compilation process, the aspectj-maven-plugin plugin will automatically weave the code with aspect annotations into the generated classes file.
When to use to?
When a method wants to add a try catch to save exception information, the LogException annotation can be used. When there is a method that does not allow errors, the NoException annotation can be used.
What is the result when adding this annotation?
Use the LogException annotation to automatically capture exceptions into the Log file, and the code can be more concise. Use the NoException annotation to automatically capture the exception to the Log file and exit the program when an exception occurs.
will get String index out of range exception when use error datetime values like '2020-02-01'
before:
MySQL [test]> select test121.k1 from test121 where k1 != ('9102-12-');
ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: String index out of range: 8
after:
MySQL [test]> select test121.k1 from test121 where k1 = '9102-12-';
ERROR 1105 (HY000): errCode = 2, detailMessage = Incorrect datetime value: '9102-12-' in expression: k1 = '9102-12-'
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
* [enhancement](fe query schedule) use try write lock to avoid too much wait time for planner; prin acii id instead of big int
---------
Co-authored-by: yiguolei <yiguolei@gmail.com>
When compaction case, memory map offsets coming to same olap convertor which is from 0 to 0+size
but it should be continue in different pages when in one segment writer .
eg :
last block with map offset : [3, 6, 8, ... 100]
this block with map offset : [5, 10, 15 ..., 100]
the same convertor should record last offset to make later coming offset followed last offset.
so after convertor :
the current offset should [105, 110, 115, ... 200], then column writer just call append_data() to make the right offset data append pages
1. Add Http interface for query q-error
2. Fix the selectivity calculation of inner join, it would always be 0 if there is only one join condition before
The problem is an exception when doing analyze:
java.lang.IllegalStateException: exceptions :
errCode = 2, detailMessage = select list expression not produced by aggregation output (missing from GROUP BY clause?): xxx
The scenario is:
select aes_decrypt(xxx,xxx) as c0 from table group by c0;
Analyze of problem:
The direct problem is mismatched of slotref, and this mismatched due to the mismatched of parameter number of aes_decrypt function. When debuging, we can see the slotref of group column is added to ExprSubstitutionMap, but can not matching with select result columns. And this is because when substiting expr it will analyze again, so the parameter would be added twice. This will cause the mismatching of function, so it would not be substitute as a slotref, the exception would be throw.
Fix:
Add call once to adding third parameter of aes_decrypt type function. Compare the child we want to add to the last child of function. If they are the same, do not add it.
