Commit Graph

20480 Commits

Author SHA1 Message Date
084434e25c [Test](tvf) add regression tests for testing orc reader #41606 #42188 (#42120)
cherry pick from #42031 #42188

---------

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
Co-authored-by: TieweiFang <ftw2139@163.com>
2024-10-21 21:31:18 +08:00
ac3d64c010 [feature](hive)support create hive table for text format #41860 (#42195)
cherry pick from #41860

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-10-21 21:30:11 +08:00
e7229c77c8 [fix](config) increase JVM memory of BE #42052 (#42194)
bp #42052
2024-10-21 20:14:43 +08:00
7568fe41f1 [fix](test) fix unstable external p0 tests #42158 (#42192)
cherry pick from #42158
2024-10-21 18:37:30 +08:00
720a4c9f30 [Fix](Branch-2.1) fix fallback to original planer when enable_server_side_prepared_statement = false (#42156) 2024-10-21 17:46:24 +08:00
a3c1657c4b [cherry-pick](branch-2.1) check end of file when reading page (#42159)
## Proposed changes
pick pr: https://github.com/apache/doris/pull/41816
2024-10-21 17:01:04 +08:00
bbd4970ed8 [feature](jdbc catalog) support gbase jdbc catalog #41027 #41587 (#42123)
cherry pick from #41027 #41587

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2024-10-21 16:52:23 +08:00
a32ad0b1f7 [cherry-pick](branch-2.1) support reading brotli compressed parquet file (#42162)
pick pr: https://github.com/apache/doris/pull/41875
2024-10-21 16:48:09 +08:00
a150d160ea [fix](jdbc catalog) fix and add mysql and doris extremum test #41679 (#42122)
cherry pick from #41679

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2024-10-21 16:39:40 +08:00
b9e2738ee6 [Fix](orc-reader) Fix StringRef nullptr data by add checking string_values empty. #42061 (#42154)
cherry pick from #42061

Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
2024-10-21 16:26:23 +08:00
da377e98cf [Pick 2.1](inverted index) enhace test_build_index case for quickly finished jobs (#42151)
## Proposed changes

bp: #42150
2024-10-21 15:52:57 +08:00
5ba0da4a84 [fix](test) fix unstable external p0 cases #42069 (#42153)
cherry pick from #42069
2024-10-21 15:04:40 +08:00
9ac8b44d65 [bugfix](hive)Use the connected user to initialize the owner of the hive table #41876 (#42121)
cherry pick from #41876

---------

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-10-21 14:41:13 +08:00
1eea8dcb2c [enhance](thirdparty) add boritli headers in thirdparty (#42017) (#42110) (#42124)
bp  (#42017) (#42110)

---------

Co-authored-by: Socrates <suxiaogang223@icloud.com>
2024-10-21 09:51:11 +08:00
29f29f494f [chore](test) add log to find bug of unstable test (#37982) (#42129) 2024-10-19 18:00:57 +08:00
dc438649d9 [bugfix](handshake) brpc handshake should not use light pool (#42115) (#42127)
The light pool may be full. Handshake is used to check the connection
state of brpc. Should not be interfered by the thread pool logic.

---------
pick #42115

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-10-19 16:19:17 +08:00
15a7b53f59 [branch-2.1](case) fix invalid test case (#42126)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-10-19 13:54:17 +08:00
9383378f2e [Fix](Export) show export statement supports specify the catalog name #41662 (#42117)
cherry pick from #41662

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
2024-10-19 10:47:28 +08:00
d5fef266ec [fix](inverted index) Fix incorrect exception handling (#42094)
https://github.com/apache/doris/pull/41874
2024-10-19 10:45:32 +08:00
702184bd92 [fix](tvf) fix that FE OOM when using tvf #40981 (#42119)
cherry pick from #40981

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
2024-10-19 08:34:38 +08:00
bce89b65fb [Test](Orc) Add tests for exception orc files #40621 (#42118)
cherry pick from #40621

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
2024-10-19 08:34:18 +08:00
ede1253666 [fix](catalog) fix filtered database when use_meta_cache=true #41939 (#42114)
cherry pick from #41939
2024-10-19 08:33:35 +08:00
f361754a71 [fix](nereids) fix regression case:test_cte_filter_pushdown (#42103)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-19 01:33:20 +08:00
c65f1044f6 [opt](log) avoid print too much log when get partitions failed #41892 (#42116)
cherry pick from #41892
2024-10-19 00:28:49 +08:00
4cab8c5178 [fix](auth)fix show load priv bug (#41723) (#42108)
pick: https://github.com/apache/doris/pull/41723
2024-10-18 22:21:50 +08:00
4edfbb5c76 [fix](nereids)modify split_part function to fold to null literal with correct data type (#42100)
pick from master https://github.com/apache/doris/pull/41109

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-18 22:15:57 +08:00
c412de8db8 [fix](Nereids) simplify range result wrong when reference is nullable (#41356) (#42096)
pick from master #41356

if reference is nullable and simplify result is boolean literal. the
real result should be:

IF(${reference} IS NULL, NULL, ${not_null_result})
2024-10-18 22:14:21 +08:00
5db44a1b91 [fix](arrays_overlap) support arrays overlap with inverted index (#42090)
## Proposed changes
backport : https://github.com/apache/doris/pull/41286
https://github.com/apache/doris/pull/41495
Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-18 22:08:39 +08:00
dde0bf92ce [fix](inverted index) Fix incorrect usage of regexp compile_err (#41944) (#42085)
https://github.com/apache/doris/pull/41944
2024-10-18 22:06:59 +08:00
b514371524 [fix] stats npe bug (#42092)
## Proposed changes

Some processors have erased the stats information of the nodes, causing
the runtime_filter_pruner to encounter a NullPointerException.

Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-18 22:05:33 +08:00
ba9f8be9f2 [fix](nereids) Fix load failed where not set database in session (#41951) (#42082)
cherry-pick #41951

load failed where not set database in session, should use label's
database if not set database in session

LOAD LABEL test_db.label_111111 ( DATA
INFILE("hdfs://hdfs01:9000/user/") INTO TABLE `test_load_tb`) WITH
BROKER "broker" ( "username" = "user", "password" = "");

ERROR 1105 (HY000): errCode = 2, detailMessage = Current database is not
set.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-18 22:04:11 +08:00
460ff02997 [cherry-pick](branch-21)fix date_floor function return wrong result (#41948) (#42065)
## Proposed changes

cherry-pick from master https://github.com/apache/doris/pull/41948

<!--Describe your changes.-->
2024-10-18 21:54:22 +08:00
9febead0cc [fix](unstablecase) fix unstable testcase (#42091)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-10-18 21:53:28 +08:00
9b7e3fbe70 [fix](statistics)Skip analyze if the collected info is invalid. (#42028) (#42087)
backport: https://github.com/apache/doris/pull/42028
2024-10-18 20:50:53 +08:00
8409f24062 [fix](Nereids) fix fold constant by be return type mismatched (#39723)(#41164)(#41331)(#41546) (#41838)
cherry-pick: #39723 #41164 #41331 #41546 because later problem is intro by prev one, so put them together
when using fold constant by be,
the return type of substring('123456',1, 3) would changed to be text, which we want it to be 3 remove windowframe in window expression to avoid folding constant on be
2024-10-18 20:34:03 +08:00
03136baacf [fix](scanner) Fix incorrect _max_thread_num in scanner context when many queries are running. #41273 (#42016)
cherry pick from #41273
2024-10-18 18:08:07 +08:00
e64f2e68e0 [opt](nereids) refine stats derive (#40654) (#40698) (#42050)
pick from master #40654 #40698
2024-10-18 16:18:10 +08:00
1236cfd159 [fix](Nereids) fix Is Cached is not Yes in Profile when enable_sql_cache=true (#42032) (#42034)
fix `Is  Cached` is not Yes in Profile when enable_sql_cache=true, introduced by #33262
2024-10-18 16:17:15 +08:00
28066a0854 [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view #40803 (#42019)
## Proposed changes

pr: https://github.com/apache/doris/pull/40803
commitId: d7e5d461
2024-10-18 12:10:53 +08:00
cec0458860 [branch-2.1][chore](dependencies)upgrade fe dependencies (#41142) (#42056)
## Proposed changes
upgrade commons-configuration2 to 2.11.0
upgrade logging-interceptor to 4.12.0
upgrade commons-compress to 1.27.1
upgrade jetty-bom to 9.4.56.v20240826
upgrade azure-sdk to 1.2.27

Iceberg depends on configuration2, and configuration2 relies on a newer
version of commons-lang3. However, there were significant breaking
changes in commons-lang3, which made it
incompatible.https://issues.apache.org/jira/browse/LANG-1705 As a
result, I rewrote the clone method.

(cherry picked from commit 945edf8dbffaa25c987bcefad59b6cde52772d4f)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-18 09:54:12 +08:00
1332f286a5 [chore](dependencies)upgrade some dependencies (#41901) (#42047)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

(cherry picked from commit 9cfc3f16681cd1ef5b8371e03d88b014e4c0a3a0)

#41901
2024-10-17 21:49:21 +08:00
fb12e10272 [fix](array-funcs)fix array agg func with decimal type (#40839) (#42023)
## Proposed changes
backport: (https://github.com/apache/doris/pull/40839)
Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-17 20:47:39 +08:00
5fe37c0708 [Feat](Nereids) support fold constant by fe (#40441)(#40772)(#40744)(#40745)(40820) (#41837)
cherry-pick from master
#40441 
#40772 
#40744 
#40745
#40820
2024-10-17 20:43:17 +08:00
80d7523a62 [Feat](Nereids) support use cbo rule hint #35925 #39715 #40167 #40958 (#41869)
pick: #35925 #39715 #40167 #40958
Add feat of force use/nouse cbo rule hint and fix pr

introduce
when not using this hint, cbo rules like INFER_SET_OPERATOR_DISTINCT
would generate two plans and compare their cost
and nereids optimizer would decide which is better. But when we want to
control the behavior of cbo rules we could use this force cbo rule hint
usage example
explain shape plan
select /*+ USE_CBO_RULE(INFER_SET_OPERATOR_DISTINCT) */
*
from t1
union
select * from t2;
the USE_CBO_RULE(INFER_SET_OPERATOR_DISTINCT) hint would force rule
INFER_SET_OPERATOR_DISTINCT to be used
and generate plan like, which hashAgg below union is generated by this
rule:

-- !with_hint_union_distinct --
----hashAgg[GLOBAL]
--------hashAgg[LOCAL]
----------PhysicalUnion
--------------hashAgg[LOCAL]
----------------PhysicalOlapScan[t1]
--------------hashAgg[LOCAL]
----------------PhysicalOlapScan[t2]
Hint log:
Used: INFER_SET_OPERATOR_DISTINCT
UnUsed:
SyntaxError:
When we want to force disable this rule, we could use
explain shape plan select /*+
NO_USE_CBO_RULE(INFER_SET_OPERATOR_DISTINCT) */ * from t1 union select *
from t2;
which would generate plan with this rule:

-- !with_hint_no_union_distinct --
----hashAgg[GLOBAL]
--------hashAgg[LOCAL]
----------PhysicalUnion
--------------PhysicalOlapScan[t1]
--------------PhysicalOlapScan[t2]
Hint log:
Used: NO_INFER_SET_OPERATOR_DISTINCT
UnUsed:
SyntaxError:
change sessionvariable enableNereidsRules to varType.remove
2024-10-17 20:36:03 +08:00
1245df670f [feat](nereids) adjust stats derive by delta row #39222 (2.1) (#42025)
## Proposed changes
pick #39222
wait JiBin merge updateRows

Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-17 19:54:52 +08:00
Pxl
4d04db467e [Bug](predicate) Fixed the problem that the number of rows in inlist #41824 (#41910)
pick from #41824
2024-10-17 17:13:00 +08:00
Pxl
f4d9ddcb00 [Improvement](runtime-filter) set some rf brpc request to ignore_eovercrowded #41698 (#41897)
pick from #41698
2024-10-17 16:57:26 +08:00
669f59ce5a [branch-2.1][feat](job)Implementing Job in Nereids (#41391) (#42012)
## Proposed changes

The JOB's execution SQL is currently defined by an older CUP file, which
causes some issues with lexical analysis in the new optimizer as it
doesn't pass under the old optimizer. Since the JOB's underlying
execution already uses the new optimizer, we're planning to fully
migrate to ANTLR4 for consistency.

(cherry picked from commit 334b473deb5ff2e5c29c5eedcfac95dd806ae622)

#41391
2024-10-17 16:56:36 +08:00
5806dae467 [fix](move-memtable) do not retry open streams (#41550) (#41999)
backport #41550
2024-10-17 15:56:56 +08:00
4521404849 [fix](case) test_limit result is unstable (#41938) (#41977)
pick from master #41938
2024-10-17 15:36:59 +08:00