Commit Graph

8289 Commits

Author SHA1 Message Date
b25d52b736 [feature](cast) remove some unused in functioncast and support some function in nereids (#22729)
1 ConvertImplGenericFromString do not need a template StringColumnType
2 remove timev1 in function cast
3 support time_to_sec , sec_to_time in nereids
2023-08-10 10:10:32 +08:00
919bfd73f1 [improvement](multi-catalog)add scanner isolation class loader (#22247)
Add scanner isolation class loader to make each plugin non-conflicting.
The BE will get scanner classes by JNI call and use JniClassLoader load them.
In the last version,we always get canner classes from the system class path by default,
so it cannot isolate the classes for each scanner
2023-08-10 10:02:46 +08:00
df1f67d835 [improve](insert) Support server side prepare insert stmt (#22353) 2023-08-10 09:59:17 +08:00
768088c95e [refactor](udaf) refactor call udaf function and support map type in return (#22508) 2023-08-09 22:44:07 +08:00
3b7a0a4713 [fix](cache) Fix enable sql cache lead to FE Full GC or OOM #22769 2023-08-09 19:24:04 +08:00
Pxl
89dc1f73b2 [Bug](materialized-view) make mv matched when preagg have value column predicate contained in mv'where clause (#22779)
1. make mv matched when preagg have value column predicate contained in mv
'where clause
2. fix `org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = BITMAP_UNION need input a bitmap column, but input INVALID_TYPE`
3. make the error message more detailed when create mv stmt parse failed
2023-08-09 19:17:55 +08:00
HB
5147c096ef [Enhancement] Add an API to query session information for all FEs (#20134)
Currently, Doris only has one interface for querying specific FE session information, and many times we need to know how many session information there are in the current cluster, so I added this API.

`
GET /rest/v1/session/all

{
"msg": "success",
"code": 0,
"data": {
"column_names": ["FE", "Id", "User", "Host", "Cluster", "Db", "Command", "Time", "State", "Info"],
"rows": [{
"FE": "10.14.170.23",
"User": "root",
"Command": "Sleep",
"State": "",
"Cluster": "default_cluster",
"Host": "10.81.85.89:31465",
"Time": "230",
"Id": "0",
"Info": "",
"Db": "db1"
},
{
"FE": "10.14.170.24",
"User": "root",
"Command": "Sleep",
"State": "",
"Cluster": "default_cluster",
"Host": "10.81.85.88:61465",
"Time": "460",
"Id": "1",
"Info": "",
"Db": "db1"
}]
},
"count": 2
}
`
2023-08-09 19:02:45 +08:00
e6a860fc9e [memo](nereids) record the chosen group expression in Group (#22661)
1. remember the chosen plan in group
2. set groupId after RecomputeLogicalPropertiesProcessor
2023-08-09 18:44:46 +08:00
2a13d15d20 [feature](Nereids): disable join order when the join number > 63 (#22708) 2023-08-09 17:09:38 +08:00
77d3d4e324 [fix](cache) add sql cache conf cache_result_max_data_size (#22645)
Only the maximum number of rows in sql cache cache_result_max_row_count is not enough. If a row of data is too large, FE may OOM.
2023-08-09 14:46:23 +08:00
690a519742 [fix](Nereids) disable or expansion when pipeline engine is disable (#22719) 2023-08-09 14:33:50 +08:00
9533918d4f [fix](delete) Fix parsing error of delete where date statement (#22690) 2023-08-09 12:33:03 +08:00
a8d690272f [refactor](Nereids) let topn runtime filter as PhysicalTopN's attr (#22745)
The original implement use MutableMap on PhysicalTopN.
It is easy to lose if we rewrite the plan after this processor.
The new implement use attr to indict whether use topn runtime filter
2023-08-09 12:13:21 +08:00
4608dcb2d9 [fix](agg) fix coredump caused by push down count aggregation (#22699)
fix coredump caused by push down count aggregation
2023-08-09 10:21:20 +08:00
d3baac2952 [improvement](resource-tag) Add Backend tag location check (#22670)
Add Backend tag location check.
Avoid user set a bad backend tag, cause create table and dynamic partitions failed.
For example, the default value for all backends tag is default, When setting the replication_allocation of a table, user use the following command: ALTER TABLE example_db.mysql_table SET ("replication_allocation" = "tag.location.tag1: 1");, it can set success, but tag1 is not exist, cause dynamic partition can't create.
2023-08-09 00:08:34 +08:00
7bfcee6e71 [improvement](variable) add annotations for variables (#22292) 2023-08-08 22:16:42 +08:00
b5d7e6e7d8 [improvement](stats) Add lifecycle hooks to AnalysisTask to make codes more clear (#22658) 2023-08-08 19:06:47 +08:00
a04e30d087 [Fix](Job)Fix Job schedule calculation start time (#22707)
Since we use division calculation, when the start time is not specified,
it may have a wrong deviation from our expected time.

For example, if it is the 7th minute now, the cycle is executed every two minutes.
Then it is calculated that the first execution is 8 minutes Because 7/2=3
3+1=4
But ideally we think it should be executed at the 9th minute
2023-08-08 18:30:38 +08:00
50dd318183 [style](jdbc catalog) Tidy the jdbc catalog java file directory (#22691) 2023-08-08 18:21:21 +08:00
f2dca848db [chore](Nereids): optimize to handle enforcer in MergeGroup() (#22709) 2023-08-08 16:56:34 +08:00
0f15d86c43 [fix](nereids) decimalv2 and float like type's common type should be consistant with old planner in arithmetic expr (#22654)
when both decimalv2 and float like type in the arithmetic expr, the common type is depend on roundPreciseDecimalV2Value session variable. If it's true, the common type is DecimalV2Type.SYSTEM_DEFAULT, otherwise its double type.
2023-08-08 15:22:04 +08:00
66784cef71 [Enhancement](Load) Stream Load using SQL (#22509)
This PR was originally #16940 , but it has not been updated for a long time due to the original author @Cai-Yao . At present, we will merge some of the code into the master first.

thanks @Cai-Yao @yiguolei
2023-08-08 13:49:04 +08:00
c4def9db5c [feature](Nereids): add enforcers in Group (#22660) 2023-08-08 13:39:55 +08:00
1617368ee1 [fix](planner) fix bug of push constant conjuncts through set operation node (#22695)
when pushing down constant conjunct into set operation node, we should assign the conjunct to agg node if there is one. This is consistant with pushing constant conjunct into inlineview.
2023-08-08 12:25:42 +08:00
d77b77a33f [feature](Nereids) eliminate sort that is not directly below result sink (#22550)
eliminate sort that is not directly below result sink.
TODO:
handle select c1 + c2 from (select c1, c2 from t order by c1) v;
2023-08-08 11:19:10 +08:00
e578e1e6a2 [opt](Nereids) turnoff pipeline when dml temporary (#22693)
pipeline could not work well for dml
2023-08-08 10:26:40 +08:00
36cea89c22 [Fix](planner)support delete conditions contain non-key columns and add check in analyze phase for delete. (#22673) 2023-08-07 21:49:53 +08:00
f074909d3c [opt](Nereids) disable strict consistency dml by default temporary (#22672)
TODO:
1. optimize exchange performance
2. let table sink do merge on one replica
2023-08-07 19:38:35 +08:00
d1a2473944 [Feature](broker)Support GCS (#20904) 2023-08-07 19:37:18 +08:00
9c91e80b0c [feature](Nereids): pushdown COUNT(*) through join (#22545) 2023-08-07 12:53:27 +08:00
97adbaadb9 fix full auto analyze (#22650) 2023-08-07 11:41:38 +08:00
023815a4b4 [fix](planner)runtime filter shouldn't be pushed through window function node (#22501) 2023-08-07 09:57:12 +08:00
1a8a1e5b16 [Feature](count_by_enum) support count_by_enum function (#22071)
count_by_enum(expr1, expr2, ... , exprN);

Treats the data in a column as an enumeration and counts the number of values in each enumeration. Returns the number of enumerated values for each column, and the number of non-null values versus the number of null values.
2023-08-06 16:05:14 +08:00
fce78fff92 [fix](rest)check response code when get image (#22272) 2023-08-06 10:34:10 +08:00
59723a1883 [fix](jdbc catalog) getInsertSql to use databaseProperName for column names (#22616) 2023-08-05 22:23:58 +08:00
d3b50e3b2a [BUG](date_trunc) fix date_trunc function only handle lower string (#22602)
fix date_trunc function only handle lower string
2023-08-05 12:53:13 +08:00
76da9f181d fix join count (#22619) 2023-08-05 12:19:28 +08:00
839b469879 [fix](meta) set parallel_pipeline_task_num when upgrading from 1.2 to 2.0 (#22618) 2023-08-05 11:04:39 +08:00
55d46e0ab9 [Fix](ut)Fix NPE due to lack of editlog object in unit test of production consumption model (#22625) 2023-08-04 22:32:15 +08:00
265cded7da [Fix](Planner) fix window function in aggregation (#22603)
Problem:
When window function in aggregation function, executor would report an error like: Required field 'node_type' was not present!

Example:
SELECT SUM(MAX(c1) OVER (PARTITION BY c2, c3)) FROM test_window_in_agg;

Reason:
When analyze aggregate, analytic expr (window function carrior when analyze) transfered to slot and loss message. So when
serialize to thrift package, TExpr can not determine node_type of analytic expr.

Solved:
We do not support aggregate(window function) yet. So we report an error when analyze.
2023-08-04 19:15:51 +08:00
872280135d [exec](pipeline) revert FE pipeline instance num pr (#22617)
* Revert "[fix](executor) only mysql connect to set GlobalPipelineTask (#22205)"
* Revert "[feature](executor) using fe version to set instance_num (#22047)"
2023-08-04 19:07:14 +08:00
d974af5feb [Fix](Load)Multi table plan not include task info (#22613) 2023-08-04 18:52:22 +08:00
9f92861c91 [fix](stats) Load partition stats unexpectedly (#22589)
syncLoadColStats method invoke stale method to deserialize columnstats after supporting load part stats,
2023-08-04 18:50:38 +08:00
95aa4d8631 [Feature](Export) Supports concurrently export of table data (#21911) 2023-08-04 18:50:17 +08:00
672acb8784 [fix](show-table-status) fix hive view NPE and external meta cache refresh issue (#22377) 2023-08-04 16:55:10 +08:00
dc06c486e8 [fix](compatibility) Version 1.2 upgraded to 2.0 compatible with miniload metadata (#22590) 2023-08-04 16:52:51 +08:00
56e8ad197c [improvement](stats) Reduce unnecessary SQL from full auto analyze #22583
1. Remove bunch of SQLs related to partition's information
2. Fix the duplicate SQLs submission
3. Fix bug that table's stats not get updated after system job finished
2023-08-04 15:52:25 +08:00
7d1e08eafa [Fix](Nereids) rand() and uuid() should not fold constant (#22492)
rand() and uuid() should not fold constant and we change the default value of fold constant for non-deterministic function to false.
2023-08-04 15:36:03 +08:00
ef53a27887 [fix](nereids) allow in or exits subquery in binary operator (#22391)
support subquery in binary operator like if( xx  in ( subquery ), 1, 0 )
2023-08-04 15:35:19 +08:00
d379b04b39 [fix](planner) fix bug of push conjuncts through second phase agg (#22417)
If there is a second phase agg, the output of the 1st phase agg is its intermediate tuple not the output tuple.
This pr fix it
2023-08-04 15:21:18 +08:00