Commit Graph

8289 Commits

Author SHA1 Message Date
Pxl
a96adc01aa [Chore](function) refactor of quantile_state (#23862)
refactor of quantile_state
2023-09-06 15:39:19 +08:00
3047d7dd07 [fix](Nereids) fix or to in rule (#23940)
or expression context can't propagation cross or expression.

for example:
```
select (a = 1 or a = 2 or a = 3) + (a = 4 or a = 5 or a = 6)
= select a in [1, 2, 3] + a in [4,5,6]
!= select a in [1, 2, 3] + a in [1, 2, 3, 4, 5, 6]
```
2023-09-06 14:58:20 +08:00
dc28878f0e [FIX](function) fix size function for array map (#23920)
Issue Number: close #xxx
now we use select size(map(1, 2)); which will make be core
and we can make size function handle array & map column both
2023-09-06 14:32:06 +08:00
45fd23c153 [feature](Nereids) transform outer join to anti join with is null condition (#23833)
create table t1(c1 int not null, c2 int not null) distributed by hash(c1) buckets 3 PROPERTIES ("replication_allocation" = "tag.location.default: 1");

create table t2(c1 int not null, c2 int not null) distributed by hash(c1) buckets 3 PROPERTIES ("replication_allocation" = "tag.location.default: 1");

select t2.* from t1 left outer t2 where t2.c1 is null
==>
t1 right anti t2
2023-09-06 11:31:57 +08:00
aee773502b [opt](Nereids) avoid generate some array in BindExpression (#23854) 2023-09-06 11:12:32 +08:00
728ee90462 [improvement](deploy) Forbid LocalDeployManager drop node (#23875)
Forbid LocalDeployManager drop nodes to prevent errors in the cluster.info file from causing nodes to be dropped.
2023-09-06 08:58:25 +08:00
b885ac5df3 [fix](statistics)Set default database while forwarding stmt to master. #23938
Set default database while forwarding request to master. Avoid No database selected error.
2023-09-06 08:53:30 +08:00
527f87117b [feature](nereids) support non_nullable function (#23762) 2023-09-06 08:52:47 +08:00
44bb94d5e7 [fe](default parameters) change remote_fragment_exec_timeout_ms from 5s to 30s (#23909) 2023-09-06 00:16:23 +08:00
13c9c41c1f [opt](hudi) reduce the memory usage of avro reader (#23745)
1. Reduce the number of threads reading avro logs and keep the readers in a fixed thread pool.
2. Regularly cleaning the cached resolvers in the thread local map by reflection.
2023-09-05 23:59:23 +08:00
2c60ae46c0 [enhancement](Nereids): remove useless if-condition (#23922) 2023-09-05 19:32:52 +08:00
771221a3f3 [improvement](fe-meta) check the image's meta version (#23847)
Sometimes, user may use a low version FE to read high version image, which may cause some undefined behavior
and hard to debug.
This PR throw an explicit error to notify the user
2023-09-05 19:08:29 +08:00
53265cc4f5 [Improvement](test)Add property to support manually use auto analyzer to analyze db (#23751)
This pr is mainly for test, make it easier to add regression test case for auto analyzer.
2023-09-05 17:08:17 +08:00
b5c66cddab [feat](optimizer) Auto upgrade column_statistics from 1.2 #23853
In Doris ver1.2 all the stats fields is not null, which might cause insert failure in the ver2.0. Add logic to upgrade table schema automatically
2023-09-05 15:57:23 +08:00
f3680af8c5 [fix](nereids)fix CheckPolicy not match uniqueTable in nereids (#23714)
fix CheckPolicy not match uniqueTable in nereids
2023-09-05 14:41:48 +08:00
7709fa5ea5 fix feut failed by mockit.internal.expectations.invocation.MissingInvocation: Missing 1 invocation to: (#23893)
mockit.internal.expectations.invocation.MissingInvocation: Missing 1 invocation to:
org.apache.doris.catalog.Env#isCheckpointThread()
mockit.internal.expectations.invocation.MissingInvocation: 
Missing 1 invocation to:
org.apache.doris.catalog.Env#isCheckpointThread()
Caused by: mockit.internal.expectations.invocation.ExpectationError
resolve http://43.132.222.7:8111/viewLog.html?buildId=213066&buildTypeId=Doris_Doris_FeUt

Issue Number: close #xxx
2023-09-05 14:30:52 +08:00
2bce8bbc66 [fix](spark load) not setting the file format cause null pointer exception (#16202) 2023-09-05 12:14:07 +08:00
4dac2d3b94 [Fix](Plan)StreamLoad cannot be parsed correctly when it contains complex where conditions (#23874) 2023-09-05 11:26:59 +08:00
e525e021ee [Enhancement](Load) stream tvf support csv header (#23797)
Co-authored-by: yiguolei <676222867@qq.com>
2023-09-05 11:15:45 +08:00
1d1a9e2bfc [improvement](graceful shutdown) waiting for all query finished when graceful shutdown (#23865)
In some cloud native deployment scenario, BE(especially the Compute Node BE) will be add to cluster and remove from cluster very frequently. User's query will fail if there is a fragment is running on the shutting down BE. Users could use stop_be.sh --grace, then BE will wait all running queries to stop to avoiding running query failure, but if the waiting time exceed the limit, then be will exit directly. During this period, FE will not send any queries to BE and waiting for all running queries to stop
2023-09-05 09:52:28 +08:00
2a3fc92d13 [fix](auth)fix after setting the user password to expire, changing the password again will not take effect (#23426)
Create a jack user and set the password to expire after 10 days.
`CREATE USER 'jack' IDENTIFIED BY '12345' PASSWORD_EXPIRE INTERVAL 10 DAY FAILED_LOGIN_ATTEMPTS 3 PASSWORD_LOCK_TIME 1 DAY;`

After the password has expired, reset the password.
`SET PASSWORD FOR 'jack' = PASSWORD('123');`


Log in with a new password, ERROE ` Your password has expired. To log in you must change it using a client that supports expired passwords`
2023-09-04 21:19:54 +08:00
db306a51b3 [fix](nereids) missing return value of resetLogicalProperties() (#23850)
physicalPlan.resetLogicalProperties(); will not change the origin plan but create a new plan with no logical property. So should update the plan using resetLogicalProperties()'s return value.
2023-09-04 17:37:26 +08:00
72b709d6a9 [opt](stats) split period collector from auto collector (#23622)
1. Split period analyze from auto collector
2. Analyze table incrementally by default
3. Rename StatisticsAutoAnalyzer to StatisticsAutoCollector
2023-09-04 17:04:16 +08:00
5dc46efcdf [feature](nereids) eliminate cascading outer join (#23754)
after eliminate outer join, create is-not-null predicate, and then this is-not-null predicate can be used to eliminate descendant outer join. the newly created is-not-null predicate will be eliminated in EliminateNotNull rule.
2023-09-04 16:00:18 +08:00
301a1d97e1 [fix](row-policy) fix creating row policy with forward issue (#23801)
The `CreateRowPolicyCommand` is implemented with overriding `run()` method.

So when executing `create row policy` in non-master FE, and forward it to Master FE,
it will call `execute(TUniqueId queryId)` method and go through `executeByNereids()`.
And because without `run()` method, it will do nothing and return OK.
So after `show row policy`, user will get empty result.

This PR fix it by implmenting the `run()` method but throw an Exception, so that it will
fallback to old planner, to do the creating row policy command normally.

The full implement of `run()` method should be implemented later.
This is just a tmp fix.
2023-09-04 15:19:37 +08:00
9bca52d002 [chore](idea) add back .idea dir under fe (#23821) 2023-09-04 14:01:00 +08:00
Pxl
bb3fadc5d3 [Bug](materialized-view) fix mv not match because cast and alias name (#23580)
fix mv not match because cast and alias name
2023-09-04 12:46:33 +08:00
12ca9d95b6 [opt](Nereids) use avl tree to construct continuous union operand (#23763) 2023-09-04 10:41:48 +08:00
acbd8ca185 [improvement](show backends) show backends print trash used (#23792) 2023-09-03 20:30:58 +08:00
da4213fc2a [improve](spark-load) set status fail message when spark load job status is cancelled. (#20798) 2023-09-03 11:40:09 +08:00
8510af3a7f [fix](hive external) support set hive version when create hive external table (#20622) 2023-09-03 11:16:46 +08:00
d367e3cf01 [fix](load) fix cancel load failed because Job could not be cancelled when job is finished or cancelled (#17730) 2023-09-03 10:57:38 +08:00
a664bc5fca [enhancement](jdbc catalog ) support postgresql partition table (#23744) 2023-09-03 10:52:01 +08:00
7f124cff68 [improvement] delete duplicate code in sql_scanner.flex (#19835) 2023-09-03 10:18:07 +08:00
347cceb530 [Feature](inverted index) push count on index down to scan node (#22687)
Co-authored-by: airborne12 <airborne12@gmail.com>
2023-09-02 22:24:43 +08:00
ace51358d0 [Enhance](ip)optimize priority_ network matching logic (#23784)
If the user has configured the wrong priority_network, direct startup failure to avoid users mistakenly assuming that the configuration is correct
If the user has not configured p_ n. Select only the last IP from the IPv4 list, rather than selecting from all IPs, to avoid users' servers not supporting IPv4
2023-09-02 21:24:11 +08:00
e9104765ea [fix](doc) modify the document error,about outfile (#22475) 2023-09-02 21:14:59 +08:00
45414db1ba [enhancement](table-meta) flush column unique ids for tables before 1.2 automatically (#23616) 2023-09-02 14:56:48 +08:00
de8fa2cff5 [Fix](thrift) Add fe master check in some thrift calls (#23757)
Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>
2023-09-02 14:05:31 +08:00
f1c354e0cf [improvement](colocate table) forbit change colocate table's replica allocation (#23064) 2023-09-02 13:54:25 +08:00
bbc893c953 [Enhancement](binlog) Add ModifyPartition, BatchModifyPartitions && ReplacePartitionOperationLog support (#23773) 2023-09-02 13:19:07 +08:00
228f0ac5bb [Feature](Multi-Catalog) support query doris bitmap column in external jdbc catalog (#23021) 2023-09-02 12:46:33 +08:00
68aa4867b0 [fix](map_agg) lost scale information for decimal type (#23776) 2023-09-02 08:03:33 +08:00
b3ccc53f24 [improvement](profile)Add workload group in audit log and profile (#23761) 2023-09-02 01:09:23 +08:00
6630f92878 [Enhancement](Load) stream tvf support json (#23752)
stream tvf support json

[{"id":1, "name":"ftw", "age":18}]
[{"id":2, "name":"xxx", "age":17}]
[{"id":3, "name":"yyy", "age":19}]
example:

curl -v --location-trusted -u root: -H "sql: insert into test.t1(c1, c2) select id, name from http_stream(\"format\" = \"json\", \"strip_outer_array\" = \"true\", \"read_json_by_line\" = \"true\")" -T /root/json_file.json http://127.0.0.1:8030/api/_http_stream
2023-09-02 01:09:06 +08:00
75e2bc8a25 [function](bitmap) support bitmap_to_base64 and bitmap_from_base64 (#23759) 2023-09-02 00:58:48 +08:00
295ea482a1 [improvement](log) optimize template function log for performance (#23746)
change log level to debug and use format in template function log for performance.
2023-09-01 19:02:33 +08:00
0b94eee4c7 [fix](rest)query_info returns empty rows #23595 2023-09-01 18:50:49 +08:00
797d9de192 [fix](Nereids) When col stats is Unknow, not expression should return the stats with selectivity of 1 2023-09-01 17:36:31 +08:00
e3bbba82cf [Fix](planner) fix to_date failed in create table as select (#23613)
Problem:
when create table as select using to_date function, it would failed

Example:
create table test_to_date properties('replication_num' = '1') as select to_date('20230816') as datev2;

Reason:
after release version 2.0, datev1 is disabled, but to_date function signature does not upgrade, so it failed when checking return type of to_date

Solved:
when getfunction, forbidden to_date with return type date_v1, datetime v1 also changed to datetime v2 and decimal v2 changed to decimal v3
2023-09-01 17:28:40 +08:00