database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
20,430 Commits 1 Branch 0 Tags
3fcd64366fbbeb2ab55fcb0fd98377fd1ecb6cac
Commit Graph

20430 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
morrySnow
3fcd64366f [opt](Nereids) use 1 as narrowest column when do column pruning on union (#41719) (#41975) 
pick from master #41719

just like previous PR #41548

this PR process union node to ensure not require any column from its
children when it is required by its parent with empty slot set
 2024-10-17 15:28:27 +08:00
daidai
b4875c2789 [fix](jni)fix jni use timezone_obj get timezone be core. (#41956) (#42003) 
bp #41956 

This PR #40225 try to pass time zone info from BE to JNI, and it use
`_state->timezone_obj().name()`
to get the timezone name.
But when we do some rolling upgrade of BE, it may coredump like:

```
*** SIGSEGV address not mapped to object (@0x610) received by PID 72661 (TID 73538 OR 0x7f2e898d1640) from PID 1552; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo_t*, void*) in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo_t*, void*) in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
 4# 0x00007F3070D3E520 in /lib/x86_64-linux-gnu/libc.so.6
 5# cctz::time_zone::name[abi:cxx11]() const in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
 6# doris::vectorized::JniConnector::open(doris::RuntimeState*, doris::RuntimeProfile*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/exec/jni_connector.cpp:87
 7# doris::vectorized::AvroJNIReader::init_fetch_table_schema_reader() at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/exec/format/avro/avro_jni_reader.cpp:119
 8# std::_Function_handler::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
 9# doris::WorkThreadPool::work_thread(int) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/work_thread_pool.hpp:159
10# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
11# start_thread at ./nptl/pthread_create.c:442
12# 0x00007F3070E22850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
172.20.50.206 last coredump sql: 2024-10-13 04:12:23,985 [query] 
```

This PR use another method: `_state->timezone()`, which just return a
string, instead of reading and initializing
time zone info file, to avoid potential coredump.
 2024-10-17 14:47:33 +08:00
zhangstar333
67d057a711 [cherry-pick](branch-21) fix conv function parser string failure return wrong result (#40530) (#41964) 
## Proposed changes

Issue Number: close #39618
cherry-pick from master (#40530)
 2024-10-17 14:45:46 +08:00
daidai
0b41cd2472 [fix](serde)fix the bug in DataTypeNullableSerDe.deserialize_column_from_fixed_json (#41217) (#41960) 
bp #41217 

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-10-17 14:36:01 +08:00
daidai
1c5c27eceb [Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable (#41659) (#41959) 
bp #41659 
## Proposed changes

Because ExternalTable will initialize the previously uninitialized table
when `getCachedRowCount()`, which is unnecessary. So for the
uninitialized table, we directly return -1.
This will increase the speed of our query `information_schema.tables`.
 2024-10-17 14:34:23 +08:00
zhangstar333
968e33f07e [cherry-pick](branch-21) pick (#39057) (#41352) (#41958) 
## Proposed changes

pick from master (#39057) (#41352)

<!--Describe your changes.-->

---------

Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com>
 2024-10-17 14:30:40 +08:00
Socrates
1b901f6fcc [cherry-pick](branch-2.1) add parquet tvf cases and fix some parquet bug (#41931) 
## Proposed changes
pick pr:
  https://github.com/apache/doris/pull/41683
  https://github.com/apache/doris/pull/41506
  https://github.com/apache/doris/pull/41338
  https://github.com/apache/doris/pull/39326

---------

Co-authored-by: morningman <morningman@163.com>
 2024-10-17 14:20:58 +08:00
zhangstar333
eea916e466 [cherry-pick](branch-21) should check the expr of auto range partition (#41626) (#41872) 
## Proposed changes

cherry-pick from master (#41626)

<!--Describe your changes.-->
 2024-10-17 12:45:49 +08:00
bobhan1
b8214952a1 [branch-2.1] Fix is_partial_update parameter is not set in append_block_with_partial_content() (#41865) 
https://github.com/apache/doris/pull/41439 forgets to set
`is_partial_update` parameter for `Tablet::lookup_row_key()` in
`append_block_with_partial_content()`
 2024-10-17 12:44:41 +08:00
morrySnow
3ff67350d0 [opt](Nereids) support all syntax to avoid fallback in multi-statement query (#41811) 2024-10-17 12:42:01 +08:00
seawinde
5736dc537c [fix](mtmv) Fix duplicate column name not check when create materialized view #40658 (#41822) 
## Proposed changes

pr: https://github.com/apache/doris/pull/40658
commitId: 252aeeb6
 2024-10-17 12:19:15 +08:00
morrySnow
95c0a7a8e3 [chore](planner) change decimal literal toSql as plainString (#41809) (#41976) 
pick from master #41809

for example, 0 with decimal(38,4) will return 0.0000
 2024-10-17 12:16:10 +08:00
zhangdong
5521a25392 [enhance](insertoverwrite)insert overwrite not fallback (#41799) 
- Insert overwrite on NEREIDS can automatically clean up garbage
temporary partitions after restart, which is not available on old
optimizers
- When insert fails, no longer throw nereids exceptions
 2024-10-17 12:14:17 +08:00
zhiqiang
7daf423e98 [opt](profile) Move ExecutedByFrontend to execution summary profile #41761 (#41831) 
cherry pick from #41761
 2024-10-17 11:37:48 +08:00
zzzxl
19784d420c [opt](inverted index) Improved top-N optimization by refining the sorting column check. (#39496) (#41954) 
https://github.com/apache/doris/pull/39496
 2024-10-17 11:31:11 +08:00
Rayner Chen
cf2ec26bc2 [fix](catalog) should return error if try using a unknown database (#40479) (#41971) 
bp #40479
 2024-10-17 11:13:56 +08:00
morrySnow
169a12058b [chore](Variant) forbid variant type as hash join key (#41673) (#41974) 
pick from master #41673
 2024-10-17 11:06:37 +08:00
lihangyu
f98aa1d08b [Fix](Branch-2.1) fix fallback to legacy planner when set group commit in session variable (#41984) 2024-10-17 10:40:33 +08:00
lihangyu
0b6447faeb [Fix](SchemaChange) refactor variant root column iterator to make row… (#41941) 
pick #41700
 2024-10-17 10:39:07 +08:00
Jibing-Li
d04082f685 [improvement](statistics)Use min row count of all replicas as tablet/table row count. (#41894) (#41978) 
backport: https://github.com/apache/doris/pull/41894
 2024-10-16 21:45:37 +08:00
Gabriel
7d99d5fcc4 [fix](analytic) Fix data distribution after analytic operator (#41902) (#41949) 
Fix data distribution after analytic operator

pick #41902
 2024-10-16 18:41:56 +08:00
Gabriel
5bd33fc88c [pick](branch-2.1) pick #41292 #41350 #41589 #41628 #41743 #41601 #41667 #41751 (#41927) 
## Proposed changes

pick #41292 #41350 #41589 #41628 #41743 #41601 #41667 #41751

<!--Describe your changes.-->

---------

Co-authored-by: Pxl <pxl290@qq.com>
 2024-10-16 15:41:28 +08:00
Xinyi Zou
e56216211e [pick](branch-2.1) pick #40667 #40714 (#41905) 
pick
#40667
#40714

---------

Co-authored-by: wangbo <wangbo@apache.org>
 2024-10-16 14:09:03 +08:00
Xinyi Zou
95d429b7de [pick](branch-2.1) pick #41891 (#41929) 
pick #41891
 2024-10-16 13:59:46 +08:00
wuwenchi
e6545a36a3 [improvement](iceberg)Parallelize splits for count(*) for 2.1 (#41169) (#41880) 
bp: #41169
 2024-10-16 10:52:06 +08:00
lw112
e62e47700d [fix](Nereids) fixed the limit offset error pick 39316 (#41878) 2024-10-16 10:38:05 +08:00
Gabriel
b185dfcbf6 [pick](branch-2.1) pick #41676 #41740 #41857 (#41904) 
pick #41676 #41740 #41857
 2024-10-15 22:41:17 +08:00
wuwenchi
a4b7d93ded [bugfix](iceberg)add prefix for endpoint with s3 client for 2.1 (#41336) (#41877) 
bp: #41336
 2024-10-15 19:59:10 +08:00
Gabriel
b91d8e2327 [Improvement](minor) Reduce locking scope (#41845) (#41844) 
pick #41845
 2024-10-15 18:39:53 +08:00
amory
78b6157aa9 [fix](ip/variant) fix information meta (#41871) 
fix datatype information meta  for ip/variant (#41666)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-10-15 18:01:14 +08:00
Rayner Chen
5fbefa084c [opt](hive) make supported hive table error msg clearer (#41616) (#41851) 
bp #41616
 2024-10-15 17:36:27 +08:00
Rayner Chen
f3389973e0 [fix](heartbeat) fill default value for required field in TFrontendPingFrontendResult (#41609) (#41854) 
bp #41609
 2024-10-15 17:33:46 +08:00
Rayner Chen
24ceb60ac7 [fix](glue) support glue on aws (#41084) (#41855) 
bp #41084
 2024-10-15 17:33:25 +08:00
zhiqiang
abcba778ff [fix](cancel) Fix cancel msg on branch-2.1 (#41798) 
Make sure we can tell cancel reason from:
1. user cancel
2. timeout
3. others

```text
mysql [demo]>set query_timeout=1;
--------------
set query_timeout=1
--------------

Query OK, 0 rows affected (0.00 sec)

mysql [demo]>select sleep(5);
--------------
select sleep(5)
--------------

ERROR 1105 (HY000): errCode = 2, detailMessage = Timeout

mysql [demo]>select sleep(5);
--------------
select sleep(5)
--------------

^C^C -- sending "KILL QUERY 0" to server ...
^C -- query aborted
ERROR 1105 (HY000): errCode = 2, detailMessage = cancel query by user from 127.0.0.1:64208
```
 2024-10-15 17:15:05 +08:00
walter
77fbe6397a [fix](http) Remove file if downloading faile is failed #41778 (#41827) 
cherry pick from #41778
 2024-10-15 15:30:29 +08:00
Rayner Chen
736bb161b1 [ci](ga) skip scope-label (#41612) (#41848) 
bp #41612
 2024-10-15 14:02:21 +08:00
yujun
7aec6fc42b [fix](auto bucket) fix auto buckets calc using the first k partition #41675 (#41820) 
cherry pick from #41675
 2024-10-15 10:26:46 +08:00
amory
94687a2f3c [fix](array/map) fix resize impl in array/map (#41595) (#41699) 
backport: https://github.com/apache/doris/pull/41595
 2024-10-15 09:50:11 +08:00
zhangstar333
d97642e9b5 [cherry-pick](branch-21) fix tablet sink shuffle without project not match the output tuple (#40299)(#41293) (#41327) 
## Proposed changes

cherry-pick from master  (#40299)(#41293)

<!--Describe your changes.-->
 2024-10-15 00:12:23 +08:00
feiniaofeiafei
afdc6800d8 [fix](Nerieds) column prune should retain at least one column for union all (#41613) (#41771) 
cherry-pick #41613 to branch-2.1
 2024-10-15 00:09:02 +08:00
Socrates
4888c632f4 [cherry-pick](branch2.1) support escape.delim and serialization.null.format for hive text (#41684) 
## Proposed changes
pick from master:
https://github.com/apache/doris/pull/40291
 2024-10-15 00:08:23 +08:00
Jibing-Li
19be696120 [Refactor](statistics)Refactor of statistics buildConnectContext. (#41553) (#41814) 
backport: https://github.com/apache/doris/pull/41553
 2024-10-14 22:09:57 +08:00
Jibing-Li
b84e128974 [fix](statistics)Fix clear stale mv row count bug. (#41653) (#41807) 
backport: https://github.com/apache/doris/pull/41653
 2024-10-14 21:14:05 +08:00
Jibing-Li
345cc9172f [improvement](statistics)Drop table stats in unprotectDropTable so that dropDB could call it as well. (#39685) (#41801) 
backport: https://github.com/apache/doris/pull/39685
 2024-10-14 19:30:49 +08:00
airborne12
ff52e73a07 [Fix](inverted index) fix match null for inverted index #41746 (#41787) 
cherry pick from #41746
 2024-10-14 14:45:36 +08:00
Gabriel
f112af0fd2 [pick](branch-2.1) pick #41555 #41592 #38204 (#41781) 
pick #41555 #41592 #38204
 2024-10-14 14:05:08 +08:00
wuwenchi
ec0c008317 [feature](paimon)support paimon with dlf for 2.1 (#41247) (#41694) 
bp: #41247
 2024-10-13 20:04:01 +08:00
abmdocrt
629f500646 [cherry-pick](branch-2.1) Pick "[Fix](delete) Fix some delete p2 case (#37689) (#41678)" (#41757) 
Pick #37689 #41678
 2024-10-12 22:43:21 +08:00
Siyang Tang
87005aa5b2 [fix](delete) Fix potential delete job stuck util timeout if exception happend in FE DeleteJob execution (#41672) (#41765) 
pick: #41672

Fail task should also count down for the count down latch to prevent job
stuck.
 2024-10-12 22:29:10 +08:00
zy-kkk
68ae6d025e [branch-2.1][improvement](jdbc catalog) Print more errors when Jdbc Catalog fails to obtain a connection on FE (#41769) 2024-10-12 21:21:54 +08:00