Commit Graph

16652 Commits

Author SHA1 Message Date
fe082e374f [doc](fix) K8s doc modify configmap code (#30420) 2024-01-27 09:10:41 +08:00
713798d549 [feature](nereids)support mark join (#30133)
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
2024-01-27 09:09:53 +08:00
f25af15842 [Fix](Nereids) Fix lost predicate when query has a filter at the right input of the outer join (#30374)
materialized view def is as following:
>        select l_shipdate, o_orderdate, l_partkey, l_suppkey, o_orderkey  
>        from lineitem 
>        left join (select * from orders where o_orderdate = '2023-12-10' ) t2 
>        on lineitem.l_orderkey = t2.o_orderkey;
    
the query as following, should add filter `o_orderdate = '2023-12-10'` on mv when query rewrite by materialized view
>        select l_shipdate, o_orderdate, l_partkey, l_suppkey, o_orderkey 
>         from lineitem 
>         left join orders 
>        on lineitem.l_orderkey = orders.o_orderkey 
>         where o_orderdate = '2023-12-10' order by 1, 2, 3, 4, 5;
2024-01-27 09:09:02 +08:00
2befa75b9c [Rf](exec) Support build exactly not cal by ndv (#30398) 2024-01-27 09:09:02 +08:00
5e4674ab66 [fix](partial update) mishandling of exceptions in the publish phase may result in data loss (#30366) 2024-01-27 09:09:02 +08:00
4bd97b5094 [docs](update) Update 2.0.4 release (#30416) 2024-01-27 09:09:02 +08:00
8543167195 [Nereids](Variant) Implement variant type and support new sub column access method (#30348)
* [Nereids](Variant) Implement variant type in Variant and support new sub column access method

The query SELECT v["a"]["b"] from simple_var WHERE cast(v["a"]["b"] as int) = 1

1. During the binding stage, the expression element_at(var, "xxx") is transformed into a SlotReference with a specified path. This conversion is tracked in the StatementContext, where the parent slot is the primary key and the paths are secondary keys. This structure, known as subColumnSlotRefMap in the StatementContext, helps to eliminate duplicates of the same slot derived from identical paths.

2. A new rule, BindSlotWithPaths, is introduced in the analysis stage. This rule is responsible for converting slots with paths into their respective slot suppliers. To ensure that slots with paths are correctly associated with the appropriate LogicalOlapScan, an additional mapping, slotToRelation, is added to the StatementContext. This mapping links the top-level slot to its corresponding relation (i.e., LogicalOlapScan). Consequently, subsequent slots with paths can determine the correct LogicalOlapScan to merge with and modify accordingly.
2024-01-27 09:09:02 +08:00
Pxl
1c449d3204 [Improvement](priv) move check priv out of analyze (#30403)
move check priv out of analyze
2024-01-27 09:09:02 +08:00
18b5b72cd9 [improve](stack_trace) avoid print stack trace in group commit (#30399) 2024-01-27 09:08:29 +08:00
9aa3cdfa1b Change auto analyze max table width default value to 100. (#30395) 2024-01-27 09:08:29 +08:00
9aaa6ba351 [Fix](Variant) fix variant lost null info after cast_column (#30153)
This could result incorrect output in hirachinal cases

```
 sql """insert into ${table_name} values (-3, '{"a" : 1, "b" : 1.5, "c" : [1, 2, 3]}')"""
    sql """insert into  ${table_name} select -2, '{"a": 11245, "b" : [123, {"xx" : 1}], "c" : {"c" : 456, "d" : "null", "e" : 7.111}}'  as json_str
            union  all select -1, '{"a": 1123}' as json_str union all select *, '{"a" : 1234, "xxxx" : "kaana"}' as json_str from numbers("number" = "4096") limit 4096 ;"""

mysql> select v["c"] from var_rs where k = -3 or k = -2;
+----------------------+
| element_at(`v`, 'c') |
+----------------------+
| [1,2,3]              |
| []                   |
+----------------------+
2 rows in set (0.04 sec)
```
2024-01-27 09:08:29 +08:00
4bdc2cde2a [chore](Nereids) enable nereids if update from doris 1.x (#30369) 2024-01-27 09:08:29 +08:00
a954bab81a [fix](function) fix error result in time_to_sec and timediff (#30248) 2024-01-27 09:08:29 +08:00
144204fecc [Refactor](Rf) refactor the code of runtime filter (#30268) 2024-01-27 09:08:29 +08:00
09ae37dcc5 [pipelineX](localexchange) Adjust local exchange plan rule (#30393) 2024-01-27 09:08:29 +08:00
6f8c133a37 [chore] Remove unused test_show_create_catalog.out (#30290) 2024-01-27 09:07:13 +08:00
25c71e386b [minor](Cooldown) Log the old rowset id when uploading rowset to S3 #30339 2024-01-27 09:07:02 +08:00
584d73e3f9 change to rc06 2024-01-27 08:56:22 +08:00
9857a38b63 [fix](fe) Fix BackendHbResponse serialization compatibility problem (#30441)
* Similar to https://github.com/apache/doris/pull/30337

* Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade
  compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion
  less than FeMetaVersion.VERSION_121 will call `BackendHbResponse.readField`

* Introduced by https://github.com/apache/doris/pull/24314
2024-01-26 17:58:14 +08:00
64f7a5f395 [fix](fe)add from_unixtime to null_result_with_one_null_param_functions set (#30388) 2024-01-25 23:56:39 +08:00
2ae02efd46 [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog (#26769)
create catalog in local test/dev env like below:
create catalog iceberg PROPERTIES ('type'='iceberg','iceberg.catalog.type' = 'hadoop', 'warehouse' = '/export/workspace/warehouse');

we will get error: "Unrecognized 'warehouse' location format because name service is required."
2024-01-25 23:56:39 +08:00
6734fb10f9 Revert "[temporary](log) Add temporary log for debug insert into lost result (#30140)" (#30382)
This reverts commit f137d4ea4c2bb7037cad23d033df72d5765b74a7.
2024-01-25 21:39:15 +08:00
75ba0aa90a [fix](group_commit) Fix commit txn problem when replaying wal (#30361) 2024-01-25 21:37:33 +08:00
7938531d1e [feature](load) enable memtable on sink node by default (#30372) 2024-01-25 21:37:33 +08:00
90ed663eb0 [fix](move-memtable) all sinks wait stream close for load timeout (#30356) 2024-01-25 21:37:33 +08:00
007a489f46 [memoryleak](execontext) dctor should be virtual to avoid memory leak (#30359)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-01-25 21:37:33 +08:00
ce0b4fed6d [feat](log) Update RestApiExceptionHandler log level from debug to warn (#30306)
* When meeting RestApiExceptionHandler, we have insufficient information
  for finding out the call stack
2024-01-25 21:37:33 +08:00
2ecc6ed0d4 [opt](inverted index)Add RAM directory null cases (#30353) 2024-01-25 21:37:33 +08:00
4452f5df19 [doc](release) fix connector release doc (#30362) 2024-01-25 21:37:10 +08:00
3ffd5b8e9a [fix](Nereids) some special expression should not be constant (#30305) 2024-01-25 21:37:10 +08:00
4681958cf7 [improvement](replica) set replica drop write editlog (#30345)
* set replica drop write editlog

* update test
2024-01-25 21:37:10 +08:00
e8a20d8899 [fix](meta) remove default_cluster prefix in recover db log (#30350) 2024-01-25 21:33:51 +08:00
9174ac921b Fix statistics p0. (#30351) 2024-01-25 21:33:50 +08:00
92d4ce31ae [improve](routine-load) optimize error msg when failed to fetch Kafka info #30298 2024-01-25 21:33:50 +08:00
0f81ecf415 [feat](Nereids): eliminate inner join by pk fk when comparing mv (#30258) 2024-01-25 21:33:50 +08:00
0a5c375068 [fix](Nereids): when predicate contains right output, don't convert outer to anti join (#30276) 2024-01-25 21:33:50 +08:00
bb021668c9 [feat](log) Add more log about cost time for dropTable (#30284)
* In some p0 test cases we have found dropTable cost more than
  5 seconds, so add more time cost log
2024-01-25 14:32:39 +08:00
ccde65b942 [fix](Cooldown) Enhance calculate logic of _has_data_to_cooldown (#30244) (#30299) 2024-01-25 13:25:34 +08:00
f87484d6b3 [fix](Nereids) fix create array function type coercion (#30329) 2024-01-25 13:24:52 +08:00
b60a272be0 [Bug](pipeline) Fix pipeline load lose data use wrong var check order (#30341) 2024-01-25 13:24:52 +08:00
6614d40dad [bugfix](core) fix core due to send rpc and request is deconstructed (#30344)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-01-25 13:24:52 +08:00
xy
b1722231c4 [Fix](doc) Modify some default values in the documentation (#30304)
Co-authored-by: xingying01 <xingying01@corp.netease.com>
2024-01-25 13:24:52 +08:00
01c394acc2 [fix](inverted index)support merge null_bitmap during index compaction (#30326)
`null_bitmap` file is not considered in index compaction process. This will lead wrong query result when doc is contain `NULL` values.
2024-01-25 13:24:52 +08:00
081da1299a [fix](fe) Fix FrontendHbResponse serialization compatibility problem (#30337)
* Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade
  compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion
  less than FeMetaVersion.VERSION_121 will call `FrontendHbResponse.readField`

* Introduced by https://github.com/apache/doris/pull/24314
2024-01-25 13:24:52 +08:00
ff47c69bf9 [doc](fix) K8s doc modify and add pvc network docs (#30307) 2024-01-25 13:24:52 +08:00
61bab1f1c0 [Fix](inverted index) fix data size when drop inverted index (#30327) 2024-01-25 13:24:52 +08:00
Pxl
1d7d7ee9b6 [Chore](join) split out join hash map from hash map (#30280)
split out join hash map from hash map
2024-01-25 13:24:52 +08:00
Pxl
5b462194d1 [Feature](materialized-view) support rewrite case when to if on legacy planner to make mv work (#30320)
support rewrite case when to if on legacy planner to make mv work
2024-01-25 13:24:52 +08:00
5ba9987d3a [Fix](executor)Release resource correctly when drop workload group (#30279) 2024-01-25 13:24:52 +08:00
3d22f9cfc8 [feature](replica) Add drop replica safely on backend (#30303) 2024-01-25 13:24:52 +08:00