Commit Graph

18522 Commits

Author SHA1 Message Date
296c343179 [improve](shcema change)fix alter table faild when modify multiple columns with column changed positions (#34244) 2024-05-10 22:11:45 +08:00
845732b440 [WIP](test) remove enable_nereids_planner in regression cases (part 3) (#34558)
previous PR:
part 1: #34417
part 2: #34490
2024-05-10 22:11:01 +08:00
9a94681b29 [refactor](type) AggStateType should not extends ScalarType (#34463)
1. let AggStateType extends Type
2. remove useless interface isFixedLengthType and supportsTablePartitioning
3. let MapType implement interface isSupported
4. let VariantType extends ScalarType
2024-05-10 22:10:42 +08:00
853dbdcb00 [Feature](PreparedStatement) implement general server side prepared (#33807) 2024-05-10 22:10:11 +08:00
6c11dd2231 [Fix](planner) fix ScalarType.getAssignmentCompatibleType() when deal boolean and decimal (#34435)
The legacy planner encounters issues when handling filters such as: c1(boolean type)=0.0(decimalv3).
The literal 0.0 is interpreted as decimalv3(1,1), and the boolean type c1 is coerced to decimalv3(1,1).
decimalv3(1,1) can only retain values in the range [0,1), while the boolean true is represented as 1, exceeding the upper bound, thus causing an overflow problem.
This pull request addresses this issue by considering the boolean type as decimalv3(1,0), making both c1 and 0.0 being cast to decimal(2,1).


Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2024-05-10 22:07:16 +08:00
087f3869fd [opt](tools) update bucket number in schema file (#34561)
update tpcds sf1000 bucket number from 64 to 32, for workarounding fdb issue under storage-compute separation arch.


Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
2024-05-10 22:06:54 +08:00
391dc35e17 [fix](Nereids) array_intersect should be a variadic function (#34543) 2024-05-10 22:06:45 +08:00
5271042a7d [bugfix](gccompile) fix gcc compile error (#34546)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-05-10 22:06:00 +08:00
7c52b9f36e [fix](txn insert) Txn insert stuck if execute on observer fe (#34547) 2024-05-10 22:05:53 +08:00
95593f8e19 [config](move-memtable) support HTTP header load_stream_per_node in stream load (#34444) 2024-05-10 22:05:35 +08:00
093fe354c8 [Improve](cache) Estimated column reader memory to control segment cache (#34526) 2024-05-10 22:05:20 +08:00
d038d3bf21 [improve](restore) Release useless info for the finished job to reduce mem usage (#34513) 2024-05-10 22:05:09 +08:00
19c20247f4 [Fix](job)Reclaim resources held by finished tasks (#34506)
closeOrReleaseResources(), which is responsible for safely closing or releasing the stmtExecutor, command, and ctx objects. The method follows a null check pattern to avoid NullPointerExceptions and ensures that resources are properly cleaned up when they are no longer needed. This improves code readability and maintains a consistent approach to resource management.
2024-05-10 22:04:49 +08:00
7c56c17ecc [Fix](nereids) fix NormalizeRepeat, change the outputExpression rewrite logic (#34196)
In NormalizeRepeat, three parts of the outputExpression of LogicalRepeat need to be pushed down and outputted by bottom project: flattenGroupingSetExpr, argumentsOfGroupingScalarFunction, argumentsOfAggregateFunction.
In the original code, use these three parts to rewrite the outputExpressions of LogicalRepeat to slots.This can cause problems in some cases, for example:
```sql
SELECT
	ROUND( SUM(pk + 1) - 3) col_alias1,
	pk + 1 AS col_alias3 
	FROM
	table_20_undef_partitions2_keys3_properties4_distributed_by53
GROUP BY
	GROUPING SETS ((pk), ()) ;
```
The three parts expression needed to be pushed down are: pk, pk+1. The original code use pk+1 to rewrite the pk + 1 AS col_alias3  to slot. But the pk+1 is not in the list of grouping outputs, and then report error.
This pr change the rewrite process,  divide the expression needed to be pushed down  into 2 parts: one is (flattenGroupingSetExpr) and the other one is (argumentsOfGroupingScalarFunction, argumentsOfAggregateFunction).
 and use the flattenGroupingSetExpr rewrite all LogicalRepeat outputExpressions, and use the argumentsOfGroupingScalarFunction, argumentsOfAggregateFunction to rewrite only the agg function arguments and the grouping scalar function.
So, in the above sql, the pk + 1 AS col_alias3  will not be rewritten to slot, and can be computed.
2024-05-10 22:03:31 +08:00
c0cca6103b [WIP](test) remove enable_nereids_planner in regression cases (part 2) (#34490) 2024-05-10 22:02:32 +08:00
89091363d9 Revert "[chore](log) add try lock db failed log (#34503)"
This reverts commit e34860b86a1cf0e5aa7afa3633fa3772230e2024.
2024-05-10 17:34:43 +08:00
435b4343d7 f 2024-05-10 16:08:52 +08:00
97bc367611 [enhancement](regression-test) modify a key type from BIGINT/LARGEINT to other type (#34436)
Co-authored-by: cjj2010 <2449402815@qq.com>
2024-05-10 14:48:52 +08:00
7bd04b43d9 [fix](tablet schedule) disable schedule need clear running tasks (#34461) 2024-05-10 14:47:02 +08:00
e34860b86a [chore](log) add try lock db failed log (#34503) 2024-05-10 14:46:49 +08:00
7b22c94255 [bugfix](compile error) unused variables compile error (#34509)
* [bugfix](compile error) unused variables compile error

* f

---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-05-10 14:46:19 +08:00
082216496e [opt](inverted index) opt for log output when matching without an index (#34024)
Generates a large volume of log output when scanning large amounts of data
2024-05-10 14:45:05 +08:00
25ae7cd65f [bug](ipv6) the ipv6 type should be uint128_t (#34121)
the ipv6 type should be uint128_t, and max value is ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
if use int128_t type, it's will be min value.
2024-05-10 14:43:46 +08:00
e2fc231b7b [refactor](move-memtable) simplify LoadStreamStub::open (#34488) 2024-05-10 14:43:31 +08:00
32cbd4a583 [chore](status) unify error code between thrift,pb, status.h (#34397)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-05-10 14:41:01 +08:00
b7c2bf81fe [chore](planner) remove some useless code (#34430)
remove vectorAnalyze and computeOutputColumn function on Expr
remove vectorOpcode and outputColumn attribute on Expr
remove useless static LOG object on some Expr class
2024-05-10 14:39:17 +08:00
9b712b03b4 [FIX]fix is_ip_address_in_range func with const param (#34266) 2024-05-10 14:37:20 +08:00
520774a24b [fix](serde) fix ipv4/v6 serde functions for arrow, orc, parquet format (#34042)
this PR is from @sjyango work in #32326,
wants merge #32326 into master branch, but it's draft and not maintain long time. so have this new PR.
Co-authored-by: sjyango <sjyang2022@zju.edu.cn>
2024-05-10 14:37:04 +08:00
Pxl
804586b342 [Improvement](sort) insert data by batch on VSortedRunMerger::get_next (#34363)
insert data by batch on VSortedRunMerger::get_next
2024-05-10 14:36:53 +08:00
cc00666be6 [opt](inverted index) add inlist condition handling to compound (#34134)
1. Previously, the compound did not support the inlist condition, which could impact performance if an inverted index was created.
2024-05-10 14:35:47 +08:00
6c30913a31 [bugfix](memtracker) should count cancelled query in wg memory (#34443)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-05-10 14:35:28 +08:00
07207b7b51 [feature](shuffle) enable strict consistency dml by default (#32958) (#34641) 2024-05-10 14:31:50 +08:00
5a3107442a [feature](tvf) support query table value function (#34516) (#34640)
This PR supports a Table Value Function called `Query`. He can push a query directly to the catalog source for execution by specifying `catalog` and `query` without parsing by Doris. Doris only receives the results returned by the query.
Currently only JDBC Catalog is supported.

Example:

```
Doris > desc function query('catalog' = 'mysql','query' = 'select count(*) as cnt from test.test');           
+-------+--------+------+------+---------+-------+
| Field | Type   | Null | Key  | Default | Extra |
+-------+--------+------+------+---------+-------+
| cnt   | BIGINT | Yes  | true | NULL    | NONE  |
+-------+--------+------+------+---------+-------+

Doris > select * from query('catalog' = 'mysql','query' = 'select count(*) as cnt from test.test');           
+----------+
| cnt      |
+----------+
| 30000000 |
+----------+
```
2024-05-10 14:29:17 +08:00
60e5583b01 [fix](nereids-Branch-2.1) fix bug: try to prune a not-exist rf #34630 2024-05-10 14:28:19 +08:00
cbe8e5c010 [opt](join) For a mark join without other conjuncts, stop probing after matching one row (#34581) 2024-05-10 13:45:34 +08:00
c055174483 [fix](insert) fix forget to abort txn when insert checkStrictMode failed (#34612) (#34632) 2024-05-10 11:29:11 +08:00
fcbdd77f78 [fix](jdbc catalog) Fix ClassLoader Scope in JdbcExecutor Initialization (#34620) 2024-05-10 11:24:39 +08:00
22da62a6d5 [fix](shuffle) compensate num rows filtered in ExchangeNode (#34605) (#34623) 2024-05-10 09:27:31 +08:00
7a40f2a547 [branch-2.1](resource)fix check available fail when s3 aws_token is set and reset as, sk faild on be. (#34219) 2024-05-09 19:06:14 +08:00
53332eb4ba [fix](catalog) refactor the schema cache for external table (#34517) (#34599)
bp #34517
2024-05-09 18:02:18 +08:00
3ae3f9d6e1 [opt](catalog) support using loading cache for db/table list in external catalog (#33610) (#34596)
bp #33610
2024-05-09 17:50:39 +08:00
e9f5388541 2.1.3-rc06 2024-05-09 00:06:18 +08:00
8fa1b78d7b Revert "[feature](shuffle) enable strict consistency dml by default (#32958)"
This reverts commit 400105a92182755bdd95a58a7d378d67c6b27f51.
2024-05-08 23:00:46 +08:00
7ec12eed9f Fix duplicate normal workload group when upgrade 2.0->2.1.1->2.1.2 (#34560) 2024-05-08 21:59:26 +08:00
77a98c0b37 add 2.1.3-rc05 2024-05-08 16:36:40 +08:00
06612b9be3 [fix](local exchange) fix bug of local exchange free blocks (#34531) 2024-05-08 16:26:51 +08:00
202cdb2744 [fix](mtmv)fix refresh failed when not use db before create MTMV (#34431) (#34522)
when create MTMV,we will save current ctl and db.

when refresh MTMV,will create an ConnectContext, and set same ctl, db to ctx

when db,ctx dropped, task will be failed.

But sometimes deleting a db does not actually have an impact, so changing it to not directly fail. If refreshing the data does cause an error, then giving the user an error message
2024-05-08 16:03:22 +08:00
e085f75a43 [opt](file-scanner) print current path when encountering error (#34365) (#34523)
bp #34365
2024-05-08 14:49:03 +08:00
400105a921 [feature](shuffle) enable strict consistency dml by default (#32958) 2024-05-08 11:00:14 +08:00
05260d6d47 [pipeline](fix) Logging blocking dependency if task is not cancelled #34502 2024-05-08 10:34:26 +08:00